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SECTION 1.0: INTRODUCTION 


1.1 Background 

The U.S. Air Force's Armstrong Laboratory at Brooks Air Force Base in San Antonio, Texas is 
researching the application of the Air Force's artificial intelligence technology in public schools 
through a series of computer-based tutors using Intelligent Tutoring Systems (ITS) technology. 

Wright Laboratory at Wright-Patterson Air Force Base (WPAFB) signed a Memorandum of 
Understanding with all of the Air Force's superlabs to support Armstrong Laboratory's research. 
Wright Laboratory then enlisted the assistance of the AUiance for Education and awarded the 
Alliance a grant to administer and implement the local component of the project. 

The Alliance for Education is a nonprofit organization which is a coalition of industry, education 
and government, acting as a third-party advocate to improve education. The Alhance for 
Education is independent of local school districts, but works closely with them. The Alliance for 
Education developed the title "Project F.A.S.T. Track" (Fundamental Academic Skills Training) 
for reference to the local component of this national project. 

Wright Laboratory originally awarded the grant to the Alliance for Education in 1992 for one year 
with a renewal option for two additional years. Delays in software development at Armstrong 
laboratory resulted in an extension of the project for one additional year. This report covers 
activities during Year Three of the project. Refer to WL-TR-94-4023 for information concerning 
Year One of the project and WL-TR-95-4005 for information concerning Year Two. 

Activities for the first year involved site selection, site preparation, teacher training and support, 
provision of technical support, public relations and program evaluation. Activities for Year Two 
involved research on an Algebra word problem solving tutor and an English reading and writing 
tutor, training additional teachers, retraining past teachers on operation of newer versions of the 
software, installation of additional equipment, provisions for technical support, public relations 
and program evaluation. 

1.2 Definition of Terms 

♦ Artificial InteUigence (Al) Technology: Computer programs that attempt to achieve 
some type of intelligent behavior 

♦ FST; (acronym for Fundamental Skills Trdning) Program developed by Armstrong 
Laboratory. 

♦ ISIS: (acronym for Instruction in Scientific Inquiry Skills), a Life Science Tutor, primary 
focus for Year Four research 

♦ Intelligent Tutoring Systems (ITS) Technology: Application of artificial intelligence that 
enhances the power of computer-based instruction by acting like an expert private tutor 

♦ R-WISE: (acronym for Reading and Writing In a Supportive Environment) Reading and 
Writing Tutor, primary focus for Year Two research 

♦ WPS: (acronym for Word Problem Solving Tutor), primary focus for Year One research 


1 



SECTION 2.0: LOCAL PROJECT OBJECTIVES 


The Alliance for Education has organized and administered the local project to meet the 
following objectives; 

I. To obtain research data on the effectiveness of the Intelligent Tutoring System (ITS) 
of the Fundamental Skills Training Program for Armstrong Laboratory hy 
establishing regional testing sites. 

♦ Primary responsibility for research design rests with the Air Force's Armstrong Laboratory 
in San Antonio, Texas. Wright Laboratory at Wright-Patterson Air Force Base signed a 
Memorandum of Understanding with all of the Air Force's superlabs to support Armstrong 
Laboratory's research. Wright Laboratory then enlisted the assistance of the Alliance for 
Education and awarded the Alliance a grant to administer and implement the local 
component of the project. 

♦ F.A.S.T. Track computer laboratories have been established at Dayton Dunbar and 
Trotwood-Madison High Schools. 

♦ 43 classes of 908 students used Dayton Dunbar and Trotwood-Madison High School 
F.A.S.T. Track computer labs an average of one day per week during the 1994-1995 
school year. 

♦ The project collected data from both Dunbar and Trotwood-Madison High Schools and 
forwarded it to Armstrong Laboratory. It included 

• Pnglish and mathematics teachers' pre-and post-experience attitude surveys 

♦ English students' on-line pre-and post-tests. 

• Mathematics students' on-line pre-and post-tests. 

♦ Mathematics and English students' on-line "journals" and "thought logs" in which 
they recorded personal observations regarding lab activities and the tutors. 

♦ In cooperation with Armstrong Laboratory, the local project research team conducted 
qualitative and quantitative research projects involving teachers and students. 

n. To deliver individualized instruction through transferring the technology of 

artificial intelligence applications to two public education systems in the Dayton 
area in 

(a) a pre-algebra word problem solving tutor 

(b) an English reading and writing tutor 

♦ Each school lab contains 28 networked computer stations for student use 

♦ In September, 1994, Armstrong Laboratory provided the Alliance with new versions of the 
WPS and R-WISE Tutors for use in the Dunbar and Trotwood-Madison High School 
F.AS.T. Track computer labs. 

♦ 28 ninth grade English and 15 mathematics classes used the F.A.S.T. Track computer labs 
an average of one day per week. 
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♦ Mathematics teachers selected curriculum "modules" (e.g., Algebraic equations, area or 
perimeter of triangles, decimals, percentages, or ratios) for each lab session. Then students 
worked through problems as their pace and in a somewhat random order so that all were 
not on the same problem at the same time. 

♦ Classes had access to all R-WISE tools, including those in which artificial intelligence was 
imbedded. Those tools included "Cubing" for idea development, "Idea Board" for 
brainstorming and outlining, and "Re-vision" for editing. 

HL To support school districts' efforts to increase student test scores on the Ohio 
Proficiency Test in (a) mathematics and (b) reading and writing. 

♦ During Year Three, local evaluators focused on the correlation between Ohio Proficiency 
Test performance and WPS/R-WISE usage. 

♦ Dayton district personnel, including administrators and teachers, credit their cohesive, 
committed Dunbar teaching staff, as well as student use of the F.A.S.T. Track Tutors with 
the significant increase in Dunbar students' scores on the Ohio Proficiency Test. 

♦ Trotwood-Madison teachers began taking Workshop class students (tenth and eleventh 
grade students who have not yet passed the mathematics portion of the Proficiency Test) to 
the F.A.S.T. Track Lab in 1993-1994 and continue doing so this year. 
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SECTION 3.0: ALLIANCE PROJECT RESPONSIBILITIES 


During Year Three, the task of the Alliance for Education was responsible for coordinating all 

facets of the local project, including the following; 

L Ensure Training and Support for Teachers and Site Coordinators 

• Arranged for four Trotwood-Madison and Dayton biology teachers, site coordinators, and a 
Dayton school district science specialist to attend the ISIS training session in San Antonio, 
Texas in August, 1995. 

• Held preliminaiy and follow-up sessions with biology teachers to supplement the training 
received in San Antonio. 

• Regularly communicated with school site coordinators and met periodically with teachers and 
district personnel to ensure adequate support. Forwarded information to Armstrong Lab. 

• Organized two joint meetings to address teacher concerns and research issues with Armstrong 
Lab personnel, English teachers, site coordinators, district supervisors and local team 
members. 

• Reimbursed Dayton and Trotwood-Madison districts for the equivalent of one class period per 
day of each site coordinator’s time to ensure adequate time to perform their duties. 

• Paid stipends to participating English and mathematics teachers and site coordinators in 
recognition of their additional responsibilities. 

n. Facilitate the Local Project Team To Implement and Oversee the Project Goals and 
Ensure Adequate Staff and Technical Support 

• Served as liaison for teachers, district administrators, and Air Force persoimel at both Wright 
Laboratory and Armstrong Laboratory to ensure research issues were addressed. 

• Contracted with the University of Dayton Research Institute for the services of Katie Thorp, 
associate research engineer, to provide technical assistance to the project and to assist the local 
evaluation team. 

• Contracted with Dr. Phillip Messner, Wright State University associate professor of 
Educational Leadership, to lead the local research team during Year Three. Also contracted 
with Dr. Messner's research assistant. Hang Pham. 

• Contracted with SelectTech Services, Inc. to ensure that qualified computer technicians were 
in the computer laboratories with teachers and students at all times for immediate resolution of 
any hardware or software problems or questions and to protect the integrity of the research. 

• Scheduled and assisted Armstrong Laboratory personnel with proctoring pre- and post-tests 
required for research, and ensured transfer of research data. 

• Submitted local status and technical reports and recommendations to Wright Laboratory 
regarding project accomplishments and future directions. 
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m. Ensure Site and Equipment Needs for Research Are Met 

• Supervised the upgrading of the monitors in both F.A.S.T. Track laboratories to accommodate 
student use of the anticipated ISIS software. 

• Requested quotes for the upgrading of servers in both F.A.S.T. Track laboratories. 

IV. Effectively Administer AU Grant Funds 

• Significant carryover funds at the end of Year Three thus requiring a smaller outlay by Wright 
Laboratory for Year Four. 

y Flexibility provided by contracting for technicians' time resulted in significant savings, 
y Costs to prepare school labs for Year Three were less than anticipated. 

^ Armstrong Lab initiated fewer meetings than anticipated for various site representatives 
which resulted in lower travel costs, 
y Publication and reporting costs were lower than anticipated, 
y Less time than anticipated was required from the technical assistant/liason and the 
education consultants. 

y A new program coordinator was hired at a lower salary in May, 1995. 
y A new control group was not used in the study as planned, therefore fewer teachers' 
stipends were necessary. 

y The WPAFB administrator was not hired as planned. 

y Additional money for Wright Connection is still available in the budget. This money 
will be used in December, 1995. 
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SECTION 4.0: YEAR THREE TIMELINE 


November 10,1994 


January 18-19,1995 


March, 1995 


May 17,1995 


May 17-18,1995 


August 7-9,1995 


August 31,1995 


September, 1995 


Program Manager and Site Coordinators attended the Point 
of Contact Meeting in Albuquerque, New Mexico to 
evaluate program's progress. 

Armstrong Laboratory personnel conducted R-WISE and 
WPS post-testing at Dunbar and Trotwood-Madison High 
Schools. 

Local project replaced VGA monitors with Super VGA 
monitors at Dayton Dunbar and Trotwood-Madison High 
Schools at the (hrection of Armstrong Laboratory personnel who 
directed that the enhanced monitors were necessary to 
accommodate the ISIS Tutor. 

Armstrong Laboratory personnel conducted a 
demonstration of the ISIS demonstration at Dunbar High 
School for eight (8) Dunbar and Trotwood-Madison 
biology teachers. 

Armstrong Laboratory personnel conducted R-WISE 
post-testing at Dunbar and Trotwood-Madison High 
Schools. 

Project trained four (4) Dunbar and Trotwood-Madison 
biology teachers in San Antonio, Texas to use the Air 
Force's ISIS Tutor and incorporate its use into their 
classroom instruction. 

Armstrong Laboratory Personnel reported the 
results of Year Two research to Dayton and 
Trotwood-Madison school district, AlUance and Wright 
Laboratory personnel, including Colonel Davis. 

New servers ordered for computer labs at Dxmbar and 
Trotwood-Madison 
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SECTION 5.0: YEAR THREE AS A MAINTENANCE YEAR 


5.1 Maintenance Year 

Due to the fact that the ISIS Tutor, originally scheduled for field-testing in local schools during 
the 1994-1995 academic year, was not ready for distribution to the schools, Armstrong 
Laboratory proposed that Year Three of the project become a maintenance yearior the R-WISE 
and WPS tutors. Armstrong Laboratory continued to collect data for English and mathematics 
classes using the R-WISE and WPS software programs, provided minor up-grades for those 
programs, and continued development of the ISIS tutor planned for delivery to schools in the fall 
of 1995. 

The local team then defined Year Three as a maintenance year in which: 

♦ There would be no major equipment purchases since only upgrades of current software 
would be installed. However, upgraded monitors were purchased in preparation for the 
Year Four specifications required for operation of the ISIS Tutor. 

♦ The local project would continue to fund full-time technical support throughout the year in 
order to ensure that the computer laboratories would be operational for Year Four research. 

♦ School districts would determine which English and mathematics classes would use the 
computer laboratories, taking into consideration Armstrong Laboratory's request for 
English classes to have top priority. 

5.2 Year Three Research Design 

5.2.1 Reading and Writing (R-WISE) Tutor 

5.2.1.1 Description of the Tutor 

The version of the R-WISE Tutor which was field tested during Year Three was designed to 
improve ninth grade English students' writing skills. The reading instructional portions of the 
tutor were omitted and the tutor was used exclusively for instruction various stages of the writing 
process including; pre-writing, drafting, revision and editing. Embedded in the tutor were guides 
that enabled students to seek help as needed for their individual writing concerns. 

The philosophy of the R-WISE computer laboratory program involves a three-step process: (1) 
classroom preparation/teaching, (2) computer lab experience and (3) classroom foUow-up/review. 

5.2.1.2 R-WISE Research Design 

There were no "non-treatment" group classes involved in the computer laboratories during Year 
Three. Instead, Armstrong Laboratory installed two versions of the R-WISE software and 
designed a series of research questions around these two versions of the tutor. The "lean" version 
provided imsolidted to the students and included many illustrated examples and models. 
The other version of the tutor was termed the "rich" version. It was slightly more complicated 
and was geared toward higher level students. The "rich" tutor provided solicited advice, which 
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students could choose to request in the form of (1) diagnosing a specific problem and selecting a 
way to repair it, (2) a mini-lesson on a specific writing skill or (3) recommended specific action. 


Initially, all of the English teachers used the "lean" version of the tutor. After a few months some 
of the teachers switched to the "rich" version. 

5.2.1.3 Armstrong Laboratory's Role in Evaluation 

As the project's primary research team, Armstrong Laboratory administered a series of tests 
including attitudinal surveys and writing samples at the beginning and end of the school year. 
Armstrong Laboratory contracted with outside evaluators to score the writing samples based on 
(1) holistic criteria which addressed the overall quality of the writing samples and (2) analytic 
criteria which addressed specific writing skills including abstraction, organization, purpose and 
development. Armstrong Laboratory agreed to disseminate their research findings to all the test 
sites when the data becomes available. 


TABLE 1: Local High School Groups Involved in Year Three Research 


SCHOOL 

SUBJECT 

TUTOR 

CLASSES 

STUDENTS 

Dunbar 

Algebra I, Part I 

WPS 

4 

67 

Dunbar 

English 9 

R-WISE 

12 

254 

Dunbar 

Language Arts 10 

R-WISE 

2 

41 

Sub-Total 



18 

362 

Trotwood 

Practical Algebra 

WPS 

4 

82 

Trotwood 

Workshop 

WPS 

7 

00 

Trotwood 

English 9 

R-WISE 

14 

346 

Sub-Total 



25 

546 

TOTAL 



43 

908 


Note: Students may have been assigned to both ninth grade English and mathematics classes 
iimmg the computer labs. Therefore, those individual students may have been counted twice. 
Algebra I, Part I is the Erst section of a two-year Algebra course. Practical Algebra is a 
one-year, less in-depth Algebra course. Workshop classes are for students in grades 10-12 who 
have not passed the Ohio Proficiency Test and require additional intervention. 
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5.2.2 Word Problem Solving (WPS) Tutor 

5.2.2.1 Description of the Tutor 

The WPS Tutor, which was initially field tested during Year One of the project, was designed to 
teach ninth-grade pre-Algebra students word problem solving skills. In addition, the tutor 
provides instruction in specific topic areas addressed in a typical pre-Algebra course. The WPS 
problem solving strategy incorporates four cognitive activities: identifying the problem, 
representing the problem, solving the problem and reflecting on the problem. 

5.2.2.2 WPS Research Design 

Pre-Algebra and Workshop classes continued to use the FST computer laboratories at both 
schools. Use of the labs was left open to the schools. Several teachers elected to continue using 
the labs with their classes, while other teachers elected for a more traditional approach. 

5.3 Local Quantitative Research Project 

In cooperation with Armstrong Laboratory and local school districts the local research team 
conducted a quantitative research project during Year Three. The purpose of this research 
activity was to determine the effect, if any, of the use of the tutors on the ability of the students to 
improve their performance on and/or pass related sections of the Ohio Proficiency Test. This test 
is a ninth-grade level test administered state-wide. Passage of the test is required to receive a 
diploma upon graduation. Data relating to students who used either of the tutors during the 
1993-1994 or 1994-1995 school years were included in this study. 

The local research team's report, entitled Fundamental Skills Tutor Correlation with Ohio 
ProScieney Test Data for the 1993-1994 & 1994-1995School Years is attached as Appendix A.. 
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SECTION 6.0: YEAR FOUR (1995-1996) 


In Year Four, the third tutor, ISIS (Instruction in Scientific Inquiry Skills), be implemented at 
Dunbar and Trotwood-Madison High Schools. The primary goal of the ISIS tutor is to teach 
students the cognitive skills underl 5 dng the principles of scientific inquiry. Specifically, students 
will understand and demonstrate how to generate a hypotheses, design an experiment to test that 
hypothesis, conduct the experiment in a simulated ecosystem, draw conclusions about the 
experiment, and accept or reject the h 3 rpothesis. 

The secondary goal of the ISIS software is to address introductory high school biology in the area 
of ecological concepts and relationships. Understanding of these concepts increase student 
potential for becoming scientifically literate, fiinctional, and critical. 

The ISIS instructional approach includes an initial skills presentation with Computer Based 
Training, student performance of particular skills within an intelligent environment, 
encyclopedia-like presentation of domain concepts within intelligent environments, and 
assignment selection based on the levels of difficulty and student proficiency. 

There are approximately 45 major domain concepts embedded in ISIS including biomes (e.g., 
grasslands, deserts, temperate deciduous forests); biotic factors (e.g., carbon dioxide, raiirfall, 
sunlight); atmospheric conditions (e.g., greenhouse effect, pollutants); and ecological relationships 
(e.g. succession, symbiosis). 

During Year Four the Alliance for Education will continue to ensure the coordination of site and 
equipment needs for successful implementation of the ISIS tutor, as well as continued use of both 
the R-WISE and the WPS tutors in the labs. 
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SECTION 7.0: NEXT STEPS 


1. In order to increase the statistical significance of the local research findings, additional 
data from the 1995-1996 school year will be added to the current statistical database. 

2. The project is currently planned for completion at the end of Year Four. Future direction 
for this project and related activities must be determined with recommendations made to 
the appropriate groups. 

3. Decisions must be made regarding placement of existing, dedicated F. A. S.T. Track 
computer equipment (Super VGA monitors, CPU's and keyboards) currently on loan to 
the schools upon conclusion of this project. 



SECTION 8.0: ADDITIONAL ACTIVITIES 


Examples of additional programs administered by the Alliance for Education and partially funded 
bytheF.AS.T Project GEMMA, Wright Connection and T*. 

• Project GEMMA (Growth in Education through a Mathematical/Scientific Mentorship 
Alliance) was designed in 1990 to provide experiences for high school to explore science and 
mathematics in the world beyond the classroom walls. Nearly 100 teachers in Montgomery and 
Greene Counties participated in summer internships at local business, industry or government 
sites where they actively took on "real world" work experiences designed to provide 
meaning&l classroom transfer opportunities. Supported by a partnership of representatives 
from Dayton area businesses, WPAFB's Wright Laboratory, schools and universities, the 
cornerstones of the project include 

♦ One-on-one business mentoring 

♦ Seminars 

♦ Strategies to transfer lessons learned to the classroom 

♦ Dissemination activities including site visitations for additional teachers and school-year 
symposiums. 

• Project Wright Connection is an expansion of Project GEMMA and began in 1995 when the 
Alliance was awarded over $2 million from the National Science Foundation. A partnership 
between the Alliance, Wright-Patterson Air Force Base's Wright Laboratory, area businesses, 
the Engineering and Science Foundation and Miami Valley schools. Project Wright 
Connection now provides opportunities for teachers in 14 counties to participate. The project 
continues to support meaningful summer internships as the cornerstone of learning. Many of 
the placements are at WPAFB. 

Grants for classroom activities through Wright Connection provide resources for educators 
to implement new techniques that address topics such as critical thinking, technology, scientific 
process and cooperative learning. Over 300 direct participants and himdreds of other school 
team members benefit from Project Wright Connection as networks are established and 
teachers gain the skiUs necessary to prepare students for the workplace of the future. 

• is an initiative to support work "with the Disney Celebration Teaching Academy, Stetson 
University and the Osceola School District to research Air Force technology applications for 
adaptation to K-12 schools. Activities involve 

♦ Ascertaining the current state of educational technology. 

♦ Identifying current technologies used in the areas of mathematics, science and 
reading/writing/communications education that can be advanced with an infusion 
of Air Force technology. 

♦ Facilitating the initial steps of the infusion process by producing a report to communicate 
the results of the partnership study. 
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EXECUTIVE SUMMARY 


Artificial intelligence based software programs have been developed by the US Air Force's 
Armstrong Laboratory to aid in teaching of the fundamental skills of writing and algebra 
word problem solving. These tutors have been field tested locally at Dayton Dunbar High 
School and Trotwood-Madison High School. The focus of this research was to determine 
the effect of tutor usage on student performance on the Ohio Proficiency Test (OPT). 

Each of the tutors was addressed separately, and in each case the study population was 
limited to students who failed the math or writing portion of the OPT in the fall. The 
comparison of subscores between groups in the fall was then used to validate the statistical 
equivalence of the groups at the beginning of the school year. This does not, however, 
mean that the OPT is a valid test to measure student learning with the tutors. 

There was a lack of any statistical significance in the passing rate on the OPT of the 
students who used the word problem solving tutor over those who did not, indicating that 
the software was not effective at helping students pass this important exam. There was, 
however, a significant improvement in passing rate for students who did not use the tutor 
in the Workshop classes at Trotwood-Madison over students in Workshop classes which 
did use the tutor. This could suggest that the tutor was not an appropriate teaching aid for 
this group of students. It could also, however, simply be a reflection of the inability of 
OPT data to reflect learning accomplished through the use of the tutor. 

Comparison of the subscores for the students who did and did not use the tutor suggested 
that the only area which showed an improvement of the tutor group over the control 
population was in the one area not believed to be addressed by the tutor (Data Analysis). 
These results further substantiate the suggestion that the word problem solving tutor is not 
an effective teaching aid of the material covered in the Ohio Proficiency Test. 

Conversely, the results for the writing tutor were much more promising. There was a 
significant increase in the passing rate for the students who used the writing tutor over 
those who did not. This strongly indicates that the writing tutor software was highly 
effective at helping students pass this important exam. 
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Comparison of the subscores for the groups shows that the tutor group was able to 
improve their relative score from the lowest to the highest in the area of Content 
Organization. Likewise, the tutor group was able to at least remain equivalent to the other 
groups in the other subscore areas. The greatest help appears to be in the area of Content 
Organization which is the area most strongly addressed by the tutor. 

Of course, other factors not controlled by the researchers can always affect the observed 
results. The fact that a large percentage of the control population came from another school 
may be a cause for concern. Likewise, differences in school curriculums, student 
populations, teacher experience and style, and mid-year updates of the tutors themselves 
could all play a part in affecting the results. 

All of this analysis must be considered with the understanding that the Ohio Proficiency 
Test was not designed to be used as a measure of instructional techniques. In addition, a 
number of variables were not controlled in this comparison and could have had a significant 
impact on the results and conclusions. Furthermore, both tutors address a variety of 
material not evaluated by the OPT which could be of value to the students. 
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1.0 BACKGROUND 


Several artificial intelligence-based software tutoring programs have been developed by the 
US Air Force’s Armstrong Laboratory to aid in teaching the fundamental skills of writing 
(R-WISE), science (ISIS), and algebra word problem solving (WPS). Each of these tutors 
has been field tested at Dayton Dunbar High School and Trotwood-Madison High School 
as well as several other sites across the nation. The design of the R-WISE and WPS mtors 
suggests they may help students increase scores and pass the Ohio Proficiency Test (OPT). 
Through correlation of student usage of the tutor with OPT results, local researchers hoped 
to determine if the tutors were indeed an advantageous tool for preparation for this 
important exam. However, it should be noted that the tutors and Ohio Proficiency Test 
were not designed for use together. A lack of an improvement in OPT scores does not 
necessarily indicate that the students did not learn as a result of using the tutor, it could 
simply mean that the OPT is not the appropriate method of measurement for the effects of 
the tutors. 


2.0 THE WORD PROBLEM SOLVING TUTOR/OPT ANALYSIS 

This analysis includes data collected fi’om Dayton Dunbar High School and Trotwood- 
Madison High School during the 1993-1994 and 1994-1995 school years. 


2.1 THE WORD PROBLEM SOLVING TUTOR 

The WPS tutor is divided into a series of modules which address various topics. These 
modules were designed to be appropriate for the class materials covered in a general 9th 
grade Algebra course. 
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2.2 THE MATH SECTION OF THE OPT 


The Ohio Department of Education has established a series of strands and learning 
outcomes which the math section of the Ohio Proficiency Test has been designed to 
measure. These strands and learning outcomes are listed in Table 1 and identified as to 
whether there are WPS tutor modules which address similar topics. Additional information 
about the mathematics learning outcomes is available in information published by the Ohio 
Department of Education. 

Examples of test results from the math section of the Ohio Proficiency Test are given in 
Table 2. Each section of the Ohio Proficiency Test is scored on three levels. The first level 
is a pass/fail marking. A student who passes the test is not given any further scoring (i.e. 
score or subscores are omitted), and only a passing mark is returned to the school. A 
student who fails the math section of the test is then given a general score which can range 
from 0 to 199 given to three significant figures (i.e. 176, 180,...). The student is then 
given a subscore ranking in each of the five following strand areas: Arithmetic, 
Measurement, Geometry, Data Analysis, and Algebra. The possible subscore rankings are 
+, *, or - as defined in Table 2. 

Through analysis of the goals and directives of the WPS tutor and the Ohio Proficiency 
Test, it was determined by the researchers that the WPS tutor did not directly address the 
issues evaluated in the Data Analysis subscore area. Therefore, the tutor would not be 
expected to increase student performance in this area. The other areas, however, were 
believed to be addressed by the tutor and should be affected by tutor usage. 


2.3 STUDY POPULATIONS 

The study population included only those students who failed the math section of the Ohio 
Proficiency Test in the fall of the year of study and for whom data was available for the 
spring test of that same year. Data were collected for the 1993-1994 school year and the 
1994-1995 school year. The sample size for each population group and year are given in 
Table 3. 
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Table 1: WPS tutor modules and OPT mathematics strands. 


OFT MATHEMATICS. STRANDS 


Arithmetic 


FOCUS OF 
WPS TUTOR 


1. Compute with whole numbers, fractions, and decimals 

2. Compare, order, and determine equivalence of 
fractions, decimals, percents, whole numbers, and integers 

3. Solve and use proportions 

4. Round numbers to the nearest thousand, hundred, ten, 
one, tenth, and hundredth 

5. Solve problems and make applications involving 
percentages 


1 . 

2. 

3. 

4. 

5. 


Measurement 


6. Select and compute with appropriate standard or metric 
units to measure length, area, volume, angles, weight, 
capacity, time, temperature, and money 

7. Convert, compare, and compute with common units of 
measure within the same measurement system 

8. Read the scale on a measurement device to the nearest 
mark and make interpolations where appropriate 


6 . 

7. 

8 . 


Geometry 


9. Recognize, classify, and use characteristics of lines and 
simple two-dimensional figures 

10. Find the perimeters (circumference) and areas of 
polygons (circles) 

11. Find surface areas and volumes of rectangular solids 


9. 

10 . 
11 . 


Yes 

No 

Yes 

Yes 

Yes 


No 

Yes 

No 


No 

Yes 

Yes 


Data Analysis 


12. Read, interpret, and use tables, charts, maps, and 
graphics to identify patterns, note trends, and draw 
conclusions 

13. Use elementary notions of probability 

14. Compute averages 


12 . 

13. 

14. 


Algebra 


No 


No 

No 


15. Solve simple number sentences and use formulas 15. Yes 

16. Evaluate dgebraic expressions (simple substitutions) 16. Yes 













Table 2: Sample results for the math section of the Ohio Proficiency Test. 


NAME 

PASS 

or 

FAIL 

SCORE 

MEASURE¬ 

MENT 

ARITH¬ 

METIC 

GEOM¬ 

ETRY 

DATA 

ANALYSIS 

ALGE¬ 

BRA 

Larry 

Student 

Pass 







Mary 

Student 

Fail 

182 

+ 

- 

- 

* 

- 

Carry 

Student 

Fail 

176 

- 

_ 

- 

+ 

* 

Berry 

Student 

Fail 

196 

+ 

+ 

- 

* 

* 


- = Performance lower than expected of students at the standard 
* = Performance approximately the same as expected of students at the standard 
+ = Performance higher than expected of students at the standard 


Table 3: Number of students included in the WPS data analysis for each group and year. 



WPS 

Control 



Dunbar 

T-M 

T-M 

Dunbar 

T-M 

T-M 



Algebra 

Algebra 

Workshop 

Algebra 

Algebra 

Workshop 

Total 

93-94 








School 

82 

55 

20 

33 

0 

0 

190 

Year 








94-95 








School 

38 

37 

72 

103 

53 

52^ 

355 

Year 








School 


■H 





ISIHI 

Total 

120 

92 

IBHII 

136 

53 

illiili 


Group 

Total 

■m 

f, 


189 

■ ' 

52 

' 

545 
























































2.3.1 CONTROL 


The control population included students who were enrolled in an algebra class which did 
not use the Air Force-developed software designed to enhance word problem solving 
proficiency. During the first year of the study, no control population was available at 
Trotwood-Madison High School. Therefore, for data comparison, the control group from 
Dunbar High School was used if needed. Regardless, the control population would 
include students who were operating at class level, above class level, and below class level. 
For the purposes of this study, the control population was assumed to be equivalent to the 
treatment population. The accuracy of this assumption can be validated by comparison of 
fall scores and subscores on the Ohio Proficiency Test. This assumption is further 
validated by the fact that the study population was limited to students who failed the math 
portion of the Ohio Proficiency Test during the fall of the year in question. 


2.3.2 WPS 

The WPS population included students from Dunbar and Trotwood-Madison High Schools 
who were enrolled in an algebra class which used the WPS tutor. This group represents 
the experimental population in this study. 


2.4 DATA ANALYSIS 

In all cases, a Chi-Square analysis with a confidence interval of 5% was used to determine 
the statistical significance of the data. This indicates that if the numbers are statistically 
significant, then there is at least a 95% certainty that the same statistics would be observed 
in other, equivalent, populations. Tabular representations of all of the data presented are 
included in Appendix Al. 

For this analysis all of the treatment groups (Dunbar Algebra I Part I, Trotwood Practical 
Algebra, and Trotwood Workshop) were combined into one large treatment group, and 
likewise their representative control groups were combined into one large control group. 
This gave the largest sample size possible and should provide the most accurate statistics. 
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Combining the groups should also show the effectiveness of using the tutor in general, 
with less affect arising from school curriculum differences, teacher styles, and class 
structure differences. 


2.4.1 COMPARISON OF DATA BETWEEN YEARS 

Comparison of grouped results by the year of study allows for validation of the equivalence 
of the study populations each year. If the study populations are equivalent, then it is 
statistically valid to combine the data from each year into one larger sample set. This then 
increases the statistical significance of further data analysis. If the groups are not 
statistically equivalent across years, the data can still be combined. However, this could 
lead to a greater spread in the data when comparisons are made between treatment and 
control groups and could enhance the hkelihood of insignificant results. 


2.4.1.1 FALL 


2.4.1.1.1 PASS/FAIL ANALYSIS 

Due to that fact that the study population was limited to students who failed the OPT in the 
fall, no comparison can be made in this area. 


2.4.1.1.2 SCORE ANALYSIS 

Due to the nearly continuous spread in the scores for the math section of the OPT, this 
analysis is complex and is not available at this time. 


2.4.1.1.3 SUBSCORE ANALYSIS 

Comparison of fall subscores between years does not show any significant difference 
between year populations in any of the subscore categories except Geometry. These data 
indicate that the 1993-1994 students had a higher competency level in Geometry than the 
1994-1995 students at the beginning of the school year. This finding was significant and 
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appeared again when the same comparison was made, with the exclusion of the Workshop 
students. The fact that all other subscore categories showed no significance supports 
combination of the data by year. However, the significance of the Geometry results should 
not be ignored. 


2.4.1.2 SPRING 

2.4.1.2.1 PASS/FAIL ANALYSIS 

Using a Chi-Square analysis to compare pass/fail rates for the spring test between the 
1993-1994 and 1994-1995 school years showed no significant difference between year 
populations. These data substantiate the validity of combining the data from these two 
years of study into one data group to enlarge the sample size. This then enhances the 
statistical validity of additional data analysis. 


2.4.1.2.2 SCORE ANALYSIS 

Due to the nearly continuous spread in the scores for the math section of the OPT, this 
analysis is complex and is not available at this time. 


2.4.1.2.3 SUBSCORE ANALYSIS 

Comparison of spring subscores between years showed that the populations were not 
statistically equivalent in the areas of Arithmetic, Data Analysis, or Algebra (Figure 1). 
However, they were statistically equivalent in the areas of Measurement and Geometry. 
During the second year of the study, student subscores in the areas of Arithmetic and 
Algebra increased significantly over the first year of the study, while students in the first 
year performed significantly better in the spring in the area of Data Analysis than did the 
students in the second year. The area of Data Analysis is not believed to be addressed by 
the tutor. Therefore, any increase in student scores should be attributed to an uncontrolled 
variable, or may suggest that the OPT is not a good measure of student achievement as it 
relates to the WPS tutor. In addition, other factors such as class variations, curriculum 
modifications, teachers reassignments, and time-on-task must also be considered. 
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Figure 1; Spring subscore analysis by year of observation for WPS analysis. Subscore 
rankings are: performed lower than expected for students at the standard (-), performed 
approximately the same as expected of students at the standard (*), or performed higher 
than expected of students at the standard (+). 


2.4.2 COMPARISON OF DATA BETWEEN GROUPS 

All of the preceding analysis suggested the students are generally statistically equivalent in 
the 1993-1994 and 1994-1995 school years. It says nothing, however, about the 
equivalence of the groups. By combining the data according to group, regardless of the 
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year of study, a comparison can be made between the groups. This, then, can be used to 
determine the effectiveness of the tutor. 


2.4.2.1 FALL 

2.4.2.1.1 PASS/FAIL ANALYSIS 

Due to the fact that the study population .was limited to students who failed the OPT in the 
fall, no comparison can be made in this area. 


2.4.2.1.2 SCORE ANALYSIS 

Due to the nearly continuous spread in the scores for the math section of the OPT, this 
analysis is complex and is not available at this time. 


2.4.2.1.3 SUBSCORE ANALYSIS 

Chi-Squared analysis of fall subscores showed no statistically significant variance in the 
groups. This indicates that the treatment and control groups were statistically equivalent at 
the beginning of the year. This enhances the validity of our comparison between these two 
groups. 


2.4.2.2 SPRING 

2.4.2.2.1 PASS/FAIL ANALYSIS 

This measure of the data gives the clearest representation of the effectiveness of the tutor at 
helping students pass the OPT. The goal of each student is to pass the test. An increase in 
score may suggest an increase in ability, but without passing the test the students and 
schools do not measure any great improvement. Figure 2 displays the percentage of 
students in the treatment and control groups passing the test during the spring of the year of 
study. These differences were ngt statistically significant. 
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WPS Control 


Figure 2: Passing percent by group for WPS analysis. The difference between the groups 
is not statistically significant. 


This suggests that use of the WPS tutor had no affect on student passing rates for the math 
section of the OPT. The lack of a difference in the passing rates could easily be attributed 
to other variables such as teacher differences, school curriculum variations, student 
population differences, and other non-controlled variables. 

Similar comparisons were also made for each of the schools and courses independently. 
These comparisons again showed a lack of significance with the exception of the 
Trotwood-Madison Workshop classes. The results for this course were statistically 
significant and are shown in Figure 3. These results suggest that the tutor may have 
lowered the chances of the students being able to pass the OPT. This could be a result of 
loss of class time due to use of the tutor, if indeed the tutor does not help. However, this 
could also simply be a result of other differences between the treatment and control 
populations and could have no basis in tutor usage at all. Since this is a localized result, 
and not strongly supported by additional statistics in the subscore analysis, it should be 
viewed with caution. 
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Figure 3: Passing percent for the Trotwood-Madison Workshop classes. 


2.4.2.2.2 SCORE ANALYSIS 

Due to the nearly continuous spread in the scores for the math section of the OPT, this 
analysis is complex and is not available at this time. 


2.4.2.2.3 SUBSCORE ANALYSIS 

The Chi-Squared analysis of spring subscores showed a statistically significant variance in 
the subscore categories of Data Analysis and Algebra (Figure 4). The other subscore 
categories did not show any significant difference between the treatment and the control 
populations. The area of Data Analysis was not believed to be addressed by the tutor. 
However, students who used the tutor appeared to do better in this subscore area. This 
could indicate that either the initial tutor goals were not properly identified, or the OPT is 
measuring another difference between the study populations. The control population 
appeared to perform better in the area of Algebra which should have been addressed by the 
tutor. This again suggests that either the tutor is not effective, or the OPT is not a good 
measure of the effect of the tutor. 
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Figure 4: Spring subscore by group for WPS analysis. Subscore rankings are: performed 
lower than expected for students at the standard (-), performed approximately the same as 
expected of students at the standard (*), or performed higher than expected of students at 
the standard (+). 

2.5 RESULTS OF WPS TUTOR ANALYSIS 

Although a lot of data were collected and analyzed, the most relevant question to be 
answered relates to the effectiveness of use of the tutor to aid students with passing the 
OPT. The fact that the study population was limited to students who failed the math 

portion of the OPT in the fall, and that the comparison of subscores between groups in the 
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fall suggest that the groups were statistically equivalent at the beginning of the school year, 
suggest that this statistical comparison between students who did and did not use the tutor 
should be valid. It does not, however, mean that the OPT is a valid test to measure student 
learning with FST. It will simply be an indication of whether students who used the tutor 
performed better on the OPT. 

The lack of any statistical significance in the passing rate of the students who used the WPS 
tutor over those who did not indicates that the WPS software may not be effective at 
helping students pass this important exam. The significant improvement in passing rate for 
students who did not use the tutor in the'Workshop classes at Trotwood-Madison over 
students in Workshop classes which did use the tutor could suggest that the tutor is not an 
appropriate teaching aid for this group of students. It could also, however, simply be a 
reflection of the inability of OPT data to reflect learning accomplished through the use of 
the tutor. 

Comparison of the subscores for the groups (Table 4) suggests that the only area which 
showed an improvement of the WPS group over the control population was in the one area 
not believed to be addressed by the tutor (Data Analysis). These results further substantiate 
the suggestion that the WPS tutor is not an effective teaching aid of the materials covered in 
the Ohio Proficiency Test. 

Table 4: Group which performed the best on a given area of the math section of the OPT. 



Fall Exam 

Spring Exam 

Arithmetic 

Equivalent 

Equivalent 

Measurement 

Equivalent 

Equivalent 

Geometry 

Equivalent 

Equivalent 

Data Analysis 

Equivalent 

WPS 

Algebra 

Equivalent 

Control 


Of course, other factors not controlled by the researchers can always affect the observed 
results. Differences in school curriculums, student populations, teacher experience and 
style, and mid-year updates of the tutors themselves could all play a part by affecting the 
data analysis. 
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3.0 THE R-WISE TTJTQR/OPT ANAT.YSTS 


This analysis includes data collected from Dayton Dunbar High School, Dayton Belmont 
High School, and Trotwood-Madison High School during the 1993-1994 and 1994-1995 
school years. 


3.1 THE R-WISE TUTOR 

The R-WISE tutor (Reading and Writing in a Supportive Environment) is divided into a 
series of tools which address various topics. Through observation and use of the tutor and 
analysis of some of the literature provided by Armstrong Laboratory on the objectives of 
the tutor, correlations were made between the writing characteristics and learning outcomes 
defined by the Ohio Department of Education for the OPT and the tutor itself. This 
comparison is shown in Table 5. 


3.2 THE WRITING SECTION OF THE OPT 

The Ohio Department of Education has established a series of characteristics and learning 
outcomes which the writing section of the Ohio Proficiency Test has been designed to 
measure. Table 5 lists the writing characteristics and related learning outcomes as defined 
by the Ohio Department of Education. 

Examples of possible test results from the writing section of the Ohio Proficiency Test are 
given in Table 6. The writing section of the OPT is scored on three levels. The first level 
is a pass/fail marking. A student who passes the test is not given any further scoring (i.e. 
score or subscores are omitted), and only a passing mark is returned to the school. A 
student who fails the writing section of the test is then given a general score which can 
range from 0 to 4.5 given in two significant figures (i.e. 3.5, 4.0,...). The student is then 
given a subscore ranking in each of the three following characteristic areas: 
Content/Organization, Language, and Writing Conventions. The possible subscore 
rankings are: satisfactory, needs some help, or needs help. 
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Table 5: OPT writing characteristics and R-WISE tutor objectives. 


OPT WRITING CHARACTERISTICS 

FOCUS OF 

R-WISE TUTOR 

Content/Organization 


1. Conveys a message related to the prompt 

1. Yes 

2. Includes supporting ideas or examples 

2. Yes 

3. Follows a logical order 

3. Yes 

4. Conveys a sense of completeness 

4. Yes 

Language 


5. Exhibits word choice appropriate to the audience, 

5. Yes 

purpose, and subject 


6. Includes clear language 

6. No 

Writing Conventions 


7. Contains complete sentences and may contain 

7. No 

purposeful fragments 


8. Exhibits subject-verb agreement 

8. No 

9. Contains standard forms of verbs and nouns 

9. No 

10. Exhibits appropriate punctuation 

10. No 

11. Exhibits appropriate capitalization 

11. No 

12. Contains correct spelling 

12. No 

13. Is legible 

13. No 
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Table 6: Sample results for the writing section of the Ohio Proficiency Test. 


NAME 

PASS 

or 

FAIL 

SCORE 

ORGANIZATION 

LANGUAGE 

WRITING 

CONVENTIONS 

Larry 

Student 

Pass 





Mary 

Student 

Fail 

4.0 

NH 

NSH 

s 

Carry 

Student 

Fail 

3.5 

NSH 

NH 

NH 

Berry 

Student 

Fail 

4.5 

NH 

NH 

S 


S = Satisfactory 

NSH = Needs Some Help 

NH = Needs Help 


Through analysis of the goals and directives of the R-WISE tutor and the Ohio Proficiency 
Test, it was determined by the researchers that the R-WISE tutor did not directly address 
the issues evaluated in the Writing Conventions subscore area. Therefore, the tutor would 
not be expected to increase student performance in this area. The other areas, however, 
were believed to be addressed by the tutor and should be affected by tutor usage. 


3.3 STUDY POPULATIONS 

The study population included only those students who failed the writing section of the 
Ohio Proficiency Test in the fall of the year of study and for whom data were available for 
the spring test of that same year. Data were collected for the 1993-1994 school year and 
the 1994-1995 school year. The sample size for each population group and year are given 
in Table 7. 





























Table 7: Number of students included in data analysis for each group and year for R-WISE 
analysis. 



R-WISE 

Dunbar 

R-WISE 

T-M 

WRITE 

Dunbar 

WRITE 

T-M 

Control 

Belmont 

Control 

T-M 

Total 

93-94 

School 

Year 

25 

44 

19 

12 

53 

42 

195 

94-95 

School 

Year 

36 

111 

0 

0 

77 

0 

224 

School 

Total 

m 

ilBlli 

illiii! 

viiiiil 

130 

42 


Group 

'Total 



iMMHl 

419 


3.3.1 CONTROL 

The control population included students who were not exposed to any Air Force- 
developed software designed to enhance writing proficiency. The majority of this 
population were students in the ninth grade at Belmont High School. However, some 
students in the control population were liinth grade students at Trotwood-Madison High 
School during the 93-94 school year who were not enrolled in a ninth grade English class 
which used the tutor. Because class rosters were not always available for classes which 
did not use the tutor, the remaining 9th grade population was used. As a result, students 
who were enrolled in a class other than a standard 9th grade English class would be 
included in this population group. This could include students who were in honors level or 
remedial level courses. In either case, the control population would include students who 
were operating at class level, above class level, and below class level. For the purposes of 
t h i s study, the control population was assumed to be equivalent to the treatment population. 
The accuracy of this assumption can be validated by comparison of fall scores and 
subscores on the Ohio Proficiency Test. This assumption is further validated by the fact 
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that the study population was limited to students who failed the writing portion of the Ohio 
Proficiency Test during the fall of the year in question. 


3.3.2 WRITE 

The WRITE population included students from Dunbar and Trotwood-Madison High 
Schools who were enrolled in a standard 9th grade English class which used a simple word 
processor to assist in writing. The software did not contain any of the tutor assistance 
available in the more advanced version of the tutor. This group was used to study the 
effect of technology usage alone compared to the use of the more advanced tutor. 


3.3.3 R-WISE 

The R-WISE population included students from Dunbar and Trotwood-Madison High 
Schools who were enrolled in a standard 9th grade English class which used the advanced 
R-WISE tutor. This group represents the true experimental population in this study. 


3.4 DATA ANALYSIS 

In all cases, a Chi-Square analysis with a confidence interval of 5% was used to determine 
the statistical significance of the data. This indicates that if the numbers are statistically 
significant, then there is at least a 95% certainty that the same statistics would be observed 
in other, equivalent, populations. Tabular representations of all of the data presented are 
included in Appendix Al. 


3.4.1 COMPARISON OF DATA BETWEEN YEARS 

Comparison of grouped results by the year of study allows for validation of the equivalence 
of the study populations each year. If the study populations are equivalent, then it is 
statistically valid to combine the data from each year into one larger sample set. This then 
increases the statistical significance of further data analysis. If the groups are not 
statistically equivalent across years, the data can still be combined. However, this could 
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lead to a greater spread in the data when comparisons are made between treatment and 
control groups and could enhance the likelihood of insignificant results. 


3.4.1.1 FALL 

3.4.1.1.1 PASS/FAIL ANALYSIS 

Due to the fact that the study population was limited to students who failed the OPT in the 
fall, no comparison can be made in this area. 


3.4.1.1.2 SCORE ANALYSIS 

Comparison of the fall scores by year is graphed in Figure 5. These data shows that the 
students scored significantly different imthe second year of the study from the first year on 
the writing section of the OPT (see Figure 5). These data suggest that the students 
performed better initially during the second year of the study than in the first. 



Figure 5: Student fall scores as a function of the year of the study for R-WISE analysis. 
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3.4.1.1.3 SUBSCORE ANALYSIS 


Comparison of fall subscores between years did not show any significant difference 
between year populations in any of the subscore categories. 


3.4.1.2 SPRING 

3.4.1.2.1 PASS/FAIL ANALYSIS 

Using a Chi-Square analysis to compare pass/fail rates for the spring test between the 
1993-1994 and 1994-1995 school years showed no significant difference between year 
populations. These data substantiate the validity of combining the data from these two 
years of study into one data group to enlarge the sample size. This then enhances the 
statistical validity of additional data analysis. 


3.4.1.2.2 SCORE ANALYSIS 

Comparison of student scores in the spring as a function of the year of the study showed 
no statistical difference between years. Therefore, although the 94-95 class appeared to 
perform better in the score analysis performed for the fall test, they were statistically 
equivalent to the 93-94 class in the spring. Since each year contained both students who 
used the tutor and those who did not, this does not necessary reflect on tutor usage. 


3.4.1.2.3 SUBSCORE ANALYSIS 

Comparison of spring subscores between years showed that the populations were not 
statistically equivalent in the areas of Content Organization and Language (Figure 6). 
However, they are statistically equivalent in the area of Writing Convention which was not 
addressed by the tutor. During the second year of the study, student subscores in the area 
of Content Organization and Language increased significantly over the first year of the 
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study. The fact that this increase was seen only in those areas addressed by the tutor and 
not in the area not addressed by the tutor suggests that it may be related to tutor usage. 



Figure 6: Spring subscore analysis by year of observation for the R-WISE analysis. 


The greater effectiveness of the tutor during the second year could be a result of software 
upgrades or greater teacher comfort, and therefore, more effective teaching. Care must be 
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taken, however, in drawing conclusions from this comparison. The ratio of the number of 
students in the control and treatment populations was not the same for the two years. 
During the 1993-1994 school year, 42% of the sample was from the control population. 
However, during the 1994-1995 school year, only 34% of the sample was from the control 
population. If it is assumed that the tutor helps students improve their subscores, then the 
larger percentage of students exposed to the tutor during the second year of the study 
would give this group an effective higher score. Of course, other factors such as class 
variations, curriculum modifications, teachers reassignments, and time-on-task should also 
be considered. 


3.4.2 COMPARISON OF DATA BETWEEN GROUPS 

All of the preceding analysis suggests the students are generally statistically equivalent in 
the 1993-1994 and 1994-1995 school years. It says nothing, however, about the 
equivalence of the groups. By combining the data according to group, regardless of the 
year of study, a comparison can be made between the groups. This, then, can be used to 
determine the effectiveness of the tutor. 


3.4.2.1 FALL 


3.4.2.1.1 PASS/FAIL ANALYSIS 

Due to the fact that the study population was limited to students who failed the OPT in the 
fall, no comparison can be made in this area. 


3.4.2.1.2 SCORE ANALYSIS 

The Chi-Squared analysis of fall scores showed a statistically significant variance in the 
data; observation of Figure 7 shows that the WRITE group performed the best, followed 
by the R-WISE and control groups. 
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Figure 7: Fall score by group for R-WISE analysis. 


3.4.2.1.3 SUBSCORE ANALYSIS 

Comparison of the fall subscores between groups showed that the groups were not 
statistically equivalent in the areas of Content Organization and Writing Conventions (See 
Figure 8). They were, however, statistically equivalent in the area of Language. 

In the area of Writing Conventions the R-WISE group appeared to perform the best, while 
in the area of Content Organization the R-WISE group appeared to perform the worst. This 
information should be kept in mind when analyzing the spring results after the R-WISE 
group has been exposed to the tutor. 
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Figure 8: Fall subscore by group for R-WISE analysis. 


3.4.2.2 SPRING 

3.4.2.2.1 PASS/FAIL ANALYSIS 

This measure of the data gives the clearest representation of the effectiveness of the tutor at 
helping students pass the OPT. The goal of each student is to pass the test. An increase in 
score may suggest an increase in ability, but without passing the test the students and 
schools do not measure any great improvement. Figure 9 displays the percentage of 
students in each group passing the test during the spring of the year of study. These 
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differences are highly statistically significant (Chi-Squared confidence level of 0.014) and 
may represent an improvement as a result of exposure to the R-WISE tutor. 



Control WRITE R-WISE 


Figure 9: Passing percent by group for R-WISE analysis. 


The increase in the passing rate for the Control group over the WRITE group suggests that 
time spent in the computer lab and away from other organized learning activities may have 
been detrimental to the students. There is a significant increase, however, measured as a 
result of the features present in the R-WISE version of the software. Again, care must be 
taken in comparison of the data, particularly in the situation of the Control group and the 
WRITE group. The difference in the passing rates is small and could easily be attributed to 
other variables such as teacher differences, school curriculum variations, student 
population differences, and other non-controlled variables. 


3.4.2.2.2 SCORE ANALYSIS 

The Chi-Squared analysis of spring scores showed a statistically significant variance in the 
data; observation of Figure 10 shows that the WRITE and R-WISE groups performed at 
about the same level and were followed by the control group. 
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In both the fall and spring analysis, the WRITE group appeared to perform the best. This 
indicates that the group started the year with a higher level of proficiency and ended the 
year in the same position. The R-WISE group was notably below the WRITE group in the 
fall analysis, but appears to be almost equivalent in the spring analysis. This suggests that 
the R-WISE group was able to significantly increase their relative scores on the exam after 
exposure to the tutor. 



Figure 10: Spring score by group for R-WISE analysis. 

It should also be noted that comparison of Figure 7 with Figure 10 shows an overall 
decrease in student scores within each group between the fall and spring testing dates. This 
is probably more of a reflection of the way the test is scored than of the students' 
performance. Students who improved their score enough to pass the test are not given a 
score or subscore in the spring. Therefore, they are absent from the spring analysis which 
is reflected in an overall lowering of the scores. 

3.4.2.2.3 SUBSCORE ANALYSIS 

Comparison of spring subscores by group showed statistical significance only in the area 
of Content Organization (see Figure 11). 
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Figure 11: Spring subscore by group for R-WISE analysis. 


In the fall subscore comparison of these two groups, they were not statistically equivalent 
in the areas of Content Organization or Writing Conventions with the R-WISE group 
performing slightly better in the area of Writing Conventions and the control group 
performing slightly better in the area of Content Organization. After exposure of the 
R-WISE and WRITE groups to computer-based training, all the groups were statistically 
equivalent in the area of Writing Conventions. This area is not believed to be addressed by 
the tutor. Therefore, if the tutor is affecting scores, we would not expect it to affect them in 
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this area. This suggests that during regular classroom instruction, the control group and 
WRITE group improved more in this area than did the R-WISE group. 

In the area of Content Organization, however, the control group initially performed better, 
while the R-WISE group performed better in the spring. This suggests that the students 
using the R-WISE tutor were able to improve their performance from significantly below 
the control group to significantly above them during the course of the year. This provides 
strong support for the ability of the R-WISE tutor to improve student performance on the 
OPT. 


3.5 RESULTS OF R-WISE DATA ANALYSIS 

Although a lot of data were collected and analyzed, the most relevant question to be 
answered relates to the effectiveness of use of the tutor to aid students with passing the 
OPT. The fact that the study population was limited to students who failed the writing 
portion of the OPT in the fall, and that the comparison of subscores between groups in the 
fall showed no significant difference between groups, suggests that the groups were 
statistically equivalent at the beginning of the school year. The increased passing rate of the 
students who used the R-WISE tutor over those who did not is significant. This strongly 
indicates that the R-WISE software may be highly effective at helping students pass this 
important exam. 

Comparison of the subscores for the two groups (Table 8) shows that the R-WISE group 
was able to improve their relative score from the lowest of the three to the highest in the 
area of Content Organization. Likewise, the R-WISE group was able to at least remain 
equivalent to the other groups in the other subscore areas. The greatest help appears to be 
in the area of Content Organization which is the area most strongly addressed by the tutor. 

Of course, other factors not controlled by the researchers can always affect the observed 
results. The fact that a large percentage of the control population came from another school 
may be a cause for concern. Likewise, differences in school curriculums, student 
populations, teacher experience and style, and mid-year updates of the tutors themselves 
could all play a part in affecting the data analysis. 
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However, the data collected to date suggest that the R-WISE software can increase the 
passing rate of students on the writing portion of the OPT by about 13%. 


Table 8: Best and worst performers on a given area of the writing section of the OPT. 



Fall.Exam. 

Spring Exam | 

Best 

Performer 

Worst 

Performer 

Best 

Performer 

Worst 

Performer 

Content Organization 

WRITE 

R-WISE 

R-WISE 

Control 

Language 

Equivalent 

Equivalent 

Writing Conventions 

R-WISE 

WRITE 

Equivalent 


4.0 CONCLUSIONS 

This analysis suggests that the tutos must be considered individually as to their merit with 
respect to being able to increase student performance on the Ohio Proficiency Test. It 
appears that the Word Problem Solving tutor aimed at 9th grade algebra students does not 
increase student performance on this important exam. It may, indeed, hinder student 
performance by utilizing valuable class time which could be better spent in alternative 
instmction. However, the R-WISE tutor designed to aid 9th grade students with the 
processes involved in writing appears to enhance student performance on the writing 
section of the Ohio Proficiency Test. Student passing rates and subscores were 
significantly impacted by exposure to the tutor. The greatest effect was measured in the 
area of Content Organization. 

All of this analysis must be considered with the understanding that the Ohio Proficiency 
Test was not designed to be used as a measure of FST instructional techniques. In 
addition, a number of variables were not controlled in this comparison and could have had 
a significant impact on the results and conclusions. Furthermore, both of the tutors address 
material which is not evaluated by the OjPT and which may of value to the students. 
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APPENDIX A1 - TABULATED DATA 

1.0 WPS TUTOR 

1.1 Comparison of Data bv Year 
1.1.1 Fall Subscore Data 


Measurement 



1993-1994- 

1994-1995 

Total 


154 

285 

439 


34 

65 

99 

+ 

2 

5 

7 

Total 

190 

355 

545 


Arithmetic 



1993-1994 

1994-1995 

Total 


155 

285 

440 

* 

32 

59 

91 

+ 

3 

11 

14 

Total 

190 

355 

545 


Geometry 



1993-1994 

1994-1995 

Total 

- 

53 

180 

233 

* 

118 

148 

266 

+ 

19 

27 

46 

Total 

190 

355 

545 


Data Analysis 



1993-1994 

1994-1995 

Total 

- 

113 

210 

323 

* 

69 

125 

194 

+ 

8 

20 

28 

Total 

190 

355 

545 
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Algebra 



1993-1994 

1994-1995 

Total 

- 

no 

228 

338 

* 

70 

119 

189 

+ 

10 

8 

18 

Total 

190 

355 

545 


1.1.2 Spring Pass/Fail Data 



1993-1994 

1994-1995 

Total 

Pass 

22 

60 

82 

Fail 

163 

295 

458 

Total 

185 

355 

540 


1.1.3 Spring Subscore Data 

Measurement 



1993-1994 

1994-1995 

Total 

- 

113 

195 

308 

* 

47 

91 

138 

+ 

3 

9 

12 

Total 

163 

295 

458 


Arithmetic 



1993-1994 

1994-1995 

Total 

- 

142 

227 

369 

* 

15 

58 

73 

+ 

5 

10 

15 

Total 

162 

295 

457 


Geometry 



1993-1994 

1994-1995 

Total 

- 

57 

132 

189 

* 

86 

135 

221 

+ 

19 

28 

47 

Total 

162 

295 

457 
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Data Analysis 



1993-1994 

1994-1995 

Total 


93 

208 

301 


62 

85 

147 

+ 

7 

2 

9 

Total 

162 

295 

457 


Algebra 



1993-1994 

1994-1995 

Total 

- 

105 

118 

223 

* 

57 

159 

216 

+ 

0 

18 

18 

Total 

162 

296 

457 


1.2 Comparison of Data bv Group 
1.2.1 Fall Subscore Data 


Measurement 



WPS 

Control 

Total 


248 

191 

439 

* 

50 

49 

99 

+ 

6 

1 

7 

Total 

304 

241 

545 


Arithmetic 



WPS 

Control 

Total 

- 

248 

192 

440 

* 

47 

44 

91 

+ 

9 

5 

14 

Total 

304 

241 

545 


Geometry 



WPS 

Control 

Total 

- 

128 

105 

233 

* 

154 

112 

266 

+ 

22 

24 

46 

Total 

304 

241 

545 





































































































Data Analysis 



WPS 

Control 

Total 


183 

140 

323 


105 

89 

194 

+ 

16 

12 

28 

Total 

304 

241 

545 


Algebra 



WPS 

Control 

Total 

- 

183 

155 

338 

* 

110 

79 

189 

+ 

11 

7 

18 

Total 

304 

241 

545 


1.2.2 Spring Pass/Fail Data 



WPS 

Control 

Total 

Pass 

39 

43 

82 

Fail 

260 

198 

458 

Total 

299 

241 

540 


1.2.3 Spring Subscore Data 

Measurement 



WPS 

Control 

Total 

- 

177 

131 

308 

* 

79 

59 

138 

+ 

4 

8 

12 

Total 

260 

198 

458 


Arithmetic 



WPS 

Control 

Total 

- 

213 

156 

369 

* 

40 

33 

73 

+ 

6 

9 

15 

Total 

259 

198 

457 


53 
























































































Geometry 



WPS 

Control 

Total 

- 

106 

83 

189 

* 

122 

99 

221 

+ 

31 

16 

47 

Total 

249 

198 

457 


Data Analysis 



WPS 

Control 

Total 

- 

160 

141 

301 


90 

57 

147 

+ 

9 

0 

9 

Total 

259 

198 

457 


Algebra 



WPS 

Control 

Total 


137 

86 

223 

* 

110 

106 

216 

+ 

12 

6 

18 

Total 

259 

198 

457 


2.0 R-WISE TUTOR 

2.1 Comparison of Data by Year 
2.1.1 Fall Score Data 



1993-1994 

1994-1995 

Total 

2 

6 

7 

13 

3 

26 

54 

80 

4 

163 

163 

326 

Total 

195 

224 

419 
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2.1.2 Fall Subscore Data 


Content Organization 



1993-1994 

1994-1995 

Total 

NH 

161 

188 

349 

NSH 

34 

36 

70 

S 

0 

0 

0 

Total 

195 

224 

419 


Language 



1993-1994 

1994-1995 

Total 

NH 

136 

141 

277 

NSH 

44 

63 

107 

S 

15 

20 

35 

Total 

195 

224 

419 


Writing Conventions 



1993-1994 

1994-1995 

Total 

NH 

120 

128 

248 

NSH 

42 

53 

95 

S 

33 

43 

76 

Total 

195 

224 

419 


2.1.3 Spring Pass/Fail Data 



1993-1994 

1994-1995 

Total 

Pass 

58 

78 

136 

Fail 

137 

146 

283 

Total 

195 

224 

419 


2.1.4 Spring Score Data 



1993-1994 

1994-1995 

Total 

2 

7 

8 

15 

3 

35 

48 

83 

4 

95 

90 

185 

Total 

137 

146 

283 


55 





























































































2.1.5 Spring Subscore Data 


Content Organization 



1993-1994 

1994-1995 

Total 

NH 

125 

106 

231 

NSH 

12 

38 

50 

S 

0 

0 

0 

Total 

137 

144 

281 


Language 



1993-1994 

1994-1995 

Total 

NH 

97 

71 

168 

NSH 

26 

48 

74 

S 

14 

25 

39 

Total 

137 

144 

281 


Writing Conventions 



1993-1994 

1994-1995 

Total 

NH 

76 

74 

150 

NSH 

29 

31 

60 

S 

32 

39 

71 

Total 

137 

144 

281 


2.2 Comparison of Data bv Group 
2.2.1 Fall Score Data 



R-WISE 

WRITE 

CONTROL 

TOTAL 

2 

2 

1 

10 

13 

3 

43 

1 

36 

80 

4 

171 

29 

126 

326 

Total 

216 

31 

172 

419 


56 





















































































2.2.2 Fall Subscore Data 


Content Organization 


NH 

NSH 

S 

Total 


R-WISE 

190 

26 

0 

216 


WRITE 

22 

9 

0 

31 


CONTROL 

137 

35 

0 

172 


TOTAL 

349 

70 

0 

419 


Language 


NH 

NSH 

S 

Total 


R-WISE 

137 

58 

21 

216 


WRITE 

20 

11 

0 

31 


CONTROL 

120 

38 

14 

172 


TOTAL 

277 

107 

35 

419 


Writing Conventions 


R-WISE 


WRITE 


CONTROL 


TOTAL 


NH 


114 


24 


110 


248 


NSH 


53 


38 


95 


49 


24 


76 


Total 


216 


31 


172 


419 


2.2.3 Spring Pass/Fail Data 


Pass 

Fail 

Total 


R-WISE 

84 

132 

216 


WRITE 

7 

24 

31 


CONTROL 

45 

127 

172 


TOTAL 

136 

283 

419 


2.2.4 Spring Score Data 



R-WISE 

WRITE 


TOTAL 

_2_ 

3 

0 

12 

15 


36 

7 

40 

83 


93 

17 

75 

185 

1 Total 

132 

24 

127 

283 





















































































































2.2.5 Spring Subscore Data 


Content Organization 



R-WISE 

WRITE 

CONTROL 

TOTAL 

NH 

99 

20 

112 

231 

NSH 

31 

4 

15 

50 

S 

0 

0 

0 

0 

Total 

130 

24 

127 

281 


Language 



R-WISE 

WRITE 

CONTROL 

TOTAL 

NH 

72 

16 

80 

168 

NSH 

37 

7 

30 

74 

S 

21 

1 

17 

39 

Total 

130 

24 

127 

281 


Writing Conventions 



R-WISE 

WRITE 

CONTROL 

TOTAL 

NH 

64 

12 

74 

150 

NSH 

29 

4 

27 

60 

S 

37 

8 

26 

71 

Total 

130 

24 

127 

281 
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1.0 BACKGROUND 


Several artificial intelligence-based software tutoring programs have been developed by the 
US Air Force's Armstrong Laboratory to aid in teaching of the fundamental skills of 
writing (R-WISE), science (ISIS), and algebra word problem solving (WPS). Each of 
these tutors has been field tested at Dayton Dunbar High School and Trotwood-Madison 
High School as well as several other sites across the nation. The design of the R-WISE 
and WPS tutors suggests they may help students pass the Ohio Proficiency Test (OPT). 
Through correlation of student usage of the tutor with OPT results, local researchers hoped 
to determine if the tutors were indeed an advantageous tool for preparation for this 
important exam. Of course, the tutors and Ohio Proficiency Test were not designed for use 
together. A lack of an improvement in OPT scores does not necessarily indicate that the 
students did not learn as a result of using the tutor, it could simply mean that the OPT is not 
the appropriate method of measurement for the tutor. 


2.0 THE WORD PROBLEM SOLVING TUTOR/QPT ANALYSIS 

This analysis includes data collected from Dayton Dunbar High School during the 1993- 
1994 and 1994-1995 school years. 


2.1 THE WORD PROBLEM SOLVING TUTOR 

The WPS tutor is divided into a series of modules which address various topics. These 
modules were designed to be appropriate for the class material covered in a general 9th 
grade Algebra course. 
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2.2 THE MATH SECTION OF THE OPT 


The Ohio Department of Education has established a series of strands and learning 
outcomes which the math section of the Ohio Proficiency Test has been designed to 
measure. These strands and learning outcomes are listed in Table A2-1 and identified as to 
whether there are WPS tutor modules which address similar topics. Additional information 
about the mathematics learning outcomes is available in information published by the Ohio 
Department of Education. 

Examples of test results from the math section of the Ohio Proficiency Test are given in 
Table A2-2. Each section of the Ohio Proficiency Test is scored on three levels. The first 
level is a pass/fail marking. A student who passes the test is not given any further scoring 
(i.e. score or subscores are omitted), and only a passing mark is returned to the school. A 
student who fails the math section of the test is then given a general score which can range 
from 0 to 199 given to three significant figures (i.e. 176, 180,...). The student is then 
given a subscore ranking in each of the five following strand areas: Arithmetic, 
Measurement, Geometry, Data Analysis, and Algebra. The possible subscore r ank ings are 
+, *, or - as defined in Table A2-2. 

Through analysis of the goals and directives of the WPS tutor and the Ohio Proficiency 
Test, it was determined by the researchers that the WPS tutor did not directly address the 
issues evaluated in the Data Analysis subscore area. Therefore, the tutor would not be 
expected to increase student performance in this area. The other areas, however, were 
believed to be addressed by the tutor and should be impacted by tutor usage. 


2.3 STUDY POPULATIONS 

The study population included only those students who failed the math section of the Ohio 
Proficiency Test in the fall of the year of study and for whom data was available for the 
spring test of that same year. Data was collected for the 1993-1994 school year and the 
1994-1995 school year. The sample size for each population group and year is given in 
Table A2-3. 
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Table A2-1: WPS tutor modules and OPT mathematics strands. 


©PT MATHEMATICS STRANDS 

FOCUS OF 
WPS TUTOR 

Arithmetic 


1. Compute with whole numbers, fractions, and decimals 

1. Yes 

2. Compare, order, and determine equivalence of 
fractions, decimals, percents, whole numbers, and integers 

2. No 

3. Solve and use proportions 

3. Yes 

4. Round numbers to the nearest thousand, hundred, ten. 

4. Yes 

one, tenth, and hundredth 

5. Solve problems and make applications involving 
percentages 

5. Yes 

Measurement 


6. Select and compute with appropriate standard or metric 
units to measure length, area, volume, angles, weight. 

6. No 

capacity, time, temperature, and money 

7. Convert, compare, and compute with common units of 

7. Yes 

measure within the same measurement system 

8. Read the scale on a measurement device to the nearest 

8. No 

mark and make interpolations where appropriate 


Geometrv 

9. Recognize, classify, and use characteristics of lines and 

9. No 

simple two-dimensional figures 

10. Find the perimeters (circumference) and areas of 
polygons (circles) 

10. Yes 

11. Find surface areas and volumes of rectangular solids 

11. Yes 

Data Analvsis 


12. Read, interpret, and use tables, charts, maps, and 
graphics to identify patterns, note trends, and draw 
conclusions 

12. No 

13. Use elementary notions of probability 

13. No 

14. Compute averages 

14. No 

Algebra 


15. Solve simple number sentences and use formulas 

15. Yes 

16. Evaluate algebraic expressions (simple substitutions) 

16. Yes 
















Table A2-2: Sample results for the math section of the Ohio Proficiency Test. 




NAME 


PASS 

or 


SCORE 


MEASURE- 

MENT 


ARITH¬ 

METIC 


GEOM- DATA 
ETRY ANALYSIS 


ALGE¬ 

BRA 


Larry 

Student 

Maiy 

Student 

Carry 

Student 

Berry 

Student 


FAIL 

Pass 


Fail 


Fail 


Fail 


182 


176 


196 


+ 


-t- 


+ 



- = Performance lower than expected of students at the standard 
* = Performance approximately the same as expected of students at the standard 
-I- = Performance higher than expected of students at the standard 


Table A2-3: Number of students included in WPS data analysis for each group and year. 



WPS 

Control 

Total 

93-94 

School 

Year 

82 

33 

115 

94-95 

School 

Year 

38 

103 

141 

Tota 


120 

136 

256 


2.3.1 CONTROL 

The control population included students who were enrolled in an algebra class which did 

not use the Air Force-developed software designed to enhance word problem solving 

proficiency. The control population would include students who were operating at class 

level, above class level, and below class level. For the purposes of this study, the control 

population was assumed to be equivalent to the treatment population. The accuracy of this 
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assumption can be validated by comparison of fall scores and subscores on the Ohio 
Proficiency Test. This assumption is further validated by the fact that the study population 
was limited to students who failed the math portion of the Ohio Proficiency Test during the 
fall of the year in question. 


2.3.2 WPS 

The WPS population included students from Dunbar High School who were enrolled in an 
algebra class which used the WPS tutor. This group represents the experimental 
population in this study. 


2.4 DATA ANALYSIS 

In all cases a Chi-Square analysis with a confidence interval of 5% was used to determine 
the statistical significance of the data. This indicates that if the numbers are statistically 
significant, then there is at least a 95% certainty that the same statistics would be observed 
in other, equivalent, populations. Tabular representations of all of the data presented are 
included in the appendix. 


2.4.1 FALL 


2.4.1.1 PASS/FAIL ANALYSIS 

Due to the fact that the study population was limited to students who failed the OPT in the 
fall, no comparison can be made in this area. 


2.4.1.2 SCORE ANALYSIS 

Due to the nearly continuous spread in the scores for the math section of the OPT, this 
analysis is complex and is not available at this time. 
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2.4.1.3 


SUBSCORE ANALYSIS 


Chi-Squared analysis of fall subscores showed no statistically significant variance in the 
groups (Figure A2-1). This indicates that the treatment and control groups were 
statistically equivalent at the beginning of the year. This enhances the validity of our 
comparison between these two groups. 



Figure A2-1: Fall subscore by group for WPS analysis. Subscore rankings are: 
performed lower than expected for students at the standard (-), performed approximately 
the same as expected of students at the standard (*), or performed higher than expected of 
students at the standard (+). The differences between the groups are not statistically 
significant. 
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2.4.2 SPRING 


2.4.2.1 PASS/FAIL ANALYSIS 

This measure of the data gives the clearest representation of the effectiveness of the tutor at 
helping students pass the OPT. The goal of each student is to pass the test. An increase in 
score may suggest an increase in ability, but without passing the test the students and 
schools do not measure any great improvement. Figure A2-2 displays the percentage of 
students in the treatment and control groups passing the test during the spring of the year of 
study. These differences are n^ statistically significant. 



Figure A2-2: Passing percent by group for WPS analysis. The difference between the 
groups is not statistically significant. 

This suggests that use of the WPS tutor had no effect on student passing rates for the math 
section of the OPT. The lack of a difference in the passing rates could easily be attributed 
to other variables such as teacher differences, student population differences, and other 
non-controlled variables. 


2.4.2.2 SCORE ANALYSIS 

Due to the nearly continuous spread in the scores for the math section of the OPT, this 
analysis is complex and is not available at this time. 
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2 . 4 . 2.3 


SUBSCORE ANALYSIS 


The Chi-Squared analysis of spring subscores showed no statistically significant variance 
in the subscore categories for any of the scored areas (Figure A2-3). 



Figure A2-3: Spring subscore by group for WPS analysis. Subscore rankings are: 
performed lower than expected for students at the standard (-), performed approximately 
the same as expected of students at the standard (*), or performed higher than expected of 
students at the standard (+). 



























The area of Data Analysis was almost statistically significant with a Chi-Square probability 
of 0.052. This would suggest that the WPS group may have performed significantly better 
than the control group in this area. However, the tutor did not appear to address the goals 
of the Data Analysis area, so any correlation between tutor use and an improvement in 
scores in this area is suspicious. This again suggests that either the tutor is not effective or 
the OPT is not a good measure of the effect of the tutor. 


2.5 RESULTS OF WPS TUTOR ANALYSIS 

Although a lot of data was collected and analyzed, the most relevant question to be 
answered relates to the effectiveness of use of the tutor to aid students with passing the 
OPT. The fact that the study population was limited to students who failed the math 
portion of the OPT in the fall, and that the comparison of subscores between groups in the 
fall suggests that the groups were statistically equivalent at the beginning of the school 
year, suggests that this statistical comparison between students who did and did not use the 
tutor should be valid. It does not, however, mean that the OPT is a valid test to measure 
student learning with the WPS tutor. It will simply be an indication of whether students 
who used the tutor performed better on the OPT. 

The lack of any statistical significance in the passing rate of the students who used the WPS 
tutor over those who did not indicates that the WPS software may not be effective at 
helping students pass this important exam. It could also, however, simply be a reflection 
of the inability of OPT data to reflect learning accomplished through the use of the tutor. 

Of course, other factors not controlled by the researchers can always affect the observed 
results. School curriculum, student populations, teacher experience and style, and mid¬ 
year updates of the tutors themselves could all play a part by affecting the data analysis. 
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3.0 THE R-WISE TUTOR/OPT ANAT.YSTS 


This analysis includes data collected from Dayton Dunbar High School and Dayton 
Belmont High School during the 1993-1994 and 1994-1995 school years. 


3.1 THE R-WISE TUTOR 

The R-WISE tutor (Reading and Writing in a Supportive Environment) is divided into a 
series of tools which address various topics. Through observation and use of the tutor and 
analysis of some of the literature provided by Armstrong Laboratory on the objectives of 
the tutor, correlations were made between the writing characteristics and lear nin g outcomes 
defined by the Ohio Department of Education for the OPT and the tutor itself. This 
comparison is shown in Table A2-4. 


3.2 THE WRITING SECTION OF THE OPT 

The Ohio Department of Education has Established a series of characteristics and lea rnin g 
outcomes which the writing section of the Ohio Proficiency Test has been designed to 
measure. Table A2-4 lists the writing characteristics and related learning outcomes as 
defined by the Ohio Department of Education. 

Examples of possible test results from the writing section of the Ohio Proficiency Test are 
given in Table A2-5. The writing section of the OPT is scored on three levels. The first 
level is a pass/fail marking. A smdent who passes the test is not given any further scoring 
(i.e. score or subscores are omitted), and only a passing mark is returned to the school. A 
student who fails the writing section of the test is then given a general score which can 
range from 0 to 4.5 given in two significant figures (i.e. 3.5,4.0,...). The student is then 
given a subscore ranking in each of the three following characteristic areas: 
Content/Organization, Language, and Writing Conventions. The possible subscore 
rankings are; satisfactory, needs some help, or needs help. 
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Table A2-4: OPT writing characteristics and R-WISE tutor objectives. 


OPT WRITING CHARACTERISTICS 

FOCUS OF 


R-WISE TUTOR 

Content/Organization 


1. Conveys a message related to the prompt 

1. Yes 

2. Includes supporting ideas or examples 

2. Yes 

3. Follows a logical order 

3. Yes 

4. Conveys a sense of completeness 

4. Yes 

Language 


5. Exhibits word choice appropriate to the audience, 

5. Yes 

purpose, and subject 


6. Includes clear language 

6. No 

Writing Conventions 


7. Contains complete sentences and may contain 

7. No 

purposeful fragments 


8. Exhibits subject-verb agreement 

8. No 

9. Contains standard forms of verbs and nouns 

9. No 

10. Exhibits appropriate punctuation 

10. No 

11. Exhibits appropriate capitalization 

11. No 

12. Contains correct spelling 

12. No 

13. Is legible 

13. No 
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Table A2-5: Sample results for the writing section of the Ohio Proficiency Test. 


NAME 

PASS 

or 

FAIL 

SCORE 

ORGANIZATION 

LANGUAGE 

WRITING 

CONVENTIONS 

Lairy 

Student 

Pass 





Maty 

Student 

Fail 

4.0 

NH 

NSH 

s 

Carry 

Student 

Fail 

3.5 

NSH 

NH 

NH 

Berry 

Student 

Fail 

4.5 

NH 

NH 

S 


S = Satisfactory 

NSH = Needs Some Help 

NH = Needs Help 

Through analysis of the goals and directives of the R-WISE tutor and the Ohio Proficiency 
Test, it was determined by the researchers that the R-WISE tutor did not directly address 
the issues evaluated in the Writing Conventions subscore area. Therefore, the tutor would 
not be expected to increase student performance in this area. The other areas, however, 
were believed to be addressed by the tutor and should be affected by tutor usage. 


3.3 STUDY POPULATIONS 

The study population included only those students who failed the writing section of the 
Ohio Proficiency Test in the fall of the year of study and for whom data were available for 
the spring test of that same year. Data were collected for the 1993-1994 school year and 
the 1994-1995 school year. The sample.size for each population group and year is given in 
Table A2-6. 
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Table A2-6: Number of students included in R-WISE data analysis for each group and 
year. 



R-WISE 

Dunbar 

WRITE 

Dunbar 

Control 

Belmont 

Total 

93-94 

School 

Year 

25 

19 

53 

97 

94-95 

School 

Year 

36 

0 

77 

113 

Total 

61 

19 

130 

210 


3.3.1 CONTROL 

The control population included students who were not exposed to any Air Force- 
developed software designed to enhance writing proficiency. The majority of this 
population were students in the ninth grade at Belmont High School. Because class rosters 
were not always available for classes which did not use the tutor, the remaining 9th grade 
population was used. As a result students who were enrolled in a class other than a 
standard 9th grade English class would be included in this population group. This could 
include students who were in honors-level or remedial-level courses. In either case the 
control population would include students who were operating at class level, above class 
level, and below class level. For the purposes of this study, the control population was 
assumed to be equivalent to the treatment population. The accuracy of this assumption can 
be validated by comparison of fall scores and subscores on the Ohio Proficiency Test. This 
assumption is further validated by the fact that the study population was limited to students 
who failed the writing portion of the Ohio Proficiency Test during the fall of the year in 
question. 
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3.3.2 WRITE 


The WRITE population included students from Dunbar High School who were enrolled in 
a standard 9th grade English class which used a simple word processor to assist in writing. 
The software did not contain any of the tutor assistance available in the more advanced 
version of the tutor. This group was used to study the effect of technology usage alone 
compared to the use of the more advanced tutor. 


3.3.3 R-WISE 

The R-WISE population included students from Dunbar High School who were enrolled in 
a standard 9th grade English class which used the advanced R-WISE tutor. This group 
represents the true experimental population in this study. 


3.4 DATA ANALYSIS 

In all cases a Chi-Square analysis with a confidence interval of 5% was used to determine 
the statistical significance of the data. This indicates that if the numbers are statistically 
significant, then there is at least a 95% certainty that the same statistics would be observed 
in other, equivalent, populations. Tabular representations of all of the data presented are 
included in the appendix. 


3.4.1 FALL 

3.4.1.1 PASS/FAIL ANALYSIS 

Due to the fact that the study population was limited to students who failed the OPT in the 
fall, no comparison can be made in this area. 
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3.4.1.2 


SCORE ANALYSIS 


The Chi-Squared analysis of fall scores showed no statistically significant variance in the 
data (Figure A2-4). This suggests that the R-WISE and control populations were 
equivalent at the beginning of each year of the study. 



I 


Figure A2-4: Fall score by group for R-WISE analysis. 

I 


3.4.1.3 SUBSCORE ANALYSIS 

Comparison of the fall subscores between groups showed that the groups were statistically 
equivalent in all of the subscore areas except Writing Conventions (See Figure A2-5). In 
this area the R-WISE and control groups appeared to perform better than the WRITE 
group. 
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Figure A2-5: Fall subscore by group for R-WISE analysis. 


3.4.2 SPRING 

3.4.2.1 PASS/FAIL ANALYSIS 

This measure of the data gives the clearest representation of the effectiveness of the tutor at 
helping students pass the OPT. The goal of each student is to pass the test. An increase in 
score may suggest an increase in ability, but without passing the test the students and 
schools do not measure any great improvement. Figure A2-6 displays the percentage of 
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students in each group passing the test during the spring of the year of study. These 
differences are not statistically significant and do not, therefore, indicate the performance of 
the tutor. 



Figure A2-6: Passing percent by group for R-WISE analysis. The differences in the 
passing rates are not statistically significant. 


3.4.2.2 SCORE ANALYSIS 

The Chi-Squared analysis of spring scores showed no statistically significant variance in 
the data. The groups scores were statistically equivalent in the fall and remained that way 
in the spring. Therefore, this analysis does not suggest that the R-WISE tutor had any 
measurable affect on student performance. 


3.4.2.3 SUBSCORE ANALYSIS 

Comparison of spring subscores by group showed no statistical significance in any of the 
subscore areas. In the fall subscore comparison of these groups, they were not statistically 
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equivalent in the area Writing Conventions, with the R-WISE and control groups 
performing better than the WRITE group. Now, after exposure of the R-WISE and 
WRITE groups to computer-based training, all the groups are statistically equivalent. This 
suggests that during regular classroom instruction, the WRITE group was able to improve 
in order to be equivalent with the other two groups. 

These data do not provide any support for the ability of the R-WISE tutor to improve 
student performance on the OPT. 


3.5 RESULTS OF R-WISE DATA ANALYSIS 


The data presented here are inconclusive as to whether the R-WISE tutor increases student 
performance on the Ohio Proficiency Test. The relatively small sample size could have an 
effect on this result as well as other factors not controlled by the researchers. The fact that 
a large percentage of the control population came from another school may be a cause for 
concern. Likewise, differences in school curriculums, student populations, teacher 
experience and style, and mid-year updates of the tutors themselves could all play a part in 
affecting the data analysis. 


4.0 CONCLUSIONS 

No significant differences in the passing rates were noted as a result of exposure to either 
the WPS or R-WISE tutors. This could be a result of either the tutor not affecting student 
performance, the Ohio Proficiency Test being a poor measure of tutor effectiveness, or too 
small of a sample size. Of course, other factors such as student population variations, 
teacher differences, and curriculum differences could also have an effect on these results 
and must always be considered. 
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APPENDIX - TABULATED DATA 


1.0 WPS TUTOR 

1.1 Fall Subscore Data 


Measurement 



WPS 

Control 

Total 

- 

106 

110 

216 

* 

13 

26 

39 

+ 

1 

0 

1 

Total 

120 

136 

256 


Arithmetic 



WPS 

Control 

Total 

- 

105 

112 

217 


12 

21 

33 

+ 

3 

3 

6 

Total 

120 

136 

256 


Geometiy 



WPS 

Control 

Total 

- 

58 

75 

133 

* 

57 

54 

111 

+ 

5 

7 

12 

Total 

120 

136 

256 


Data Analysis 



WPS 

Control 

Total 

- 

82 

85 

167 


35 

46 

81 

+ 

3 

5 

8 

Total 

120 

136 

256 


82 













































































Algebra 



WPS 

Control 

Total 


77 

88 

165 

* 

41 

44 

85 

+ 

2 

4 

6 

Total 

120 

136 

256 


1.2.2 Spring Pass/Fail Data 



WPS 

Control 

Total 

Pass 

14 

15 

29 

Fail 

106 

121 

227 

Total 

120 

136 

256 


1.2.3 Spring Subscore Data 

Measurement 



WPS 

Control 

Total 


80 

80 

160 

* 

25 

34 

59 

+ 

1 

7 

8 

Total 

106 

121 

227 


Arithmetic 



WPS 

Control 

Total 

- 

95 

106 

201 

* 

7 

9 

16 

+ 

3 

6 

9 

Total 

105 

121 

226 


Geometry 



WPS 

Control 

Total 

- 

50 

56 

106 

* 

45 

60 

105 

+ 

10 

5 

15 

Total 

105 

121 

226 
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Data Analysis 



WPS 

Control 

Total 

- 

70 

84 

154 


30 

37 

67 

+ 

5 

0 

5 

Total 

105 

121 

226 


Algebra 



WPS 

Control 

Total 

- 

64 

59 

123 

* 

38 

60 

98 

+ 

3 

2 

5 

Total 

105 

121 

226 




2.0 R-WISE TUTOR 


2.1 Fall Score Data 



R-WISE 

WRITE 

CONTROL 

TOTAL 

2 

1 

0 

7 

8 

3 

16 

1 

26 

43 

4 

45 

18 

97 

160 

Total 

62 

19 

130 

211 


2.2 Fall Subscore Data 


Content Organization 




WRITE 

CONTROL 


NH 

53 

12 

98 

163 

NSH 

9 

7 

32 

48 

S 

0 

0 

0 

0 

Total 

62 

19 

130 

211 


Language 



R-WISE 

WRITE 

CONTROL 

TOTAL 

NH 

45 

12 

92 

149 

NSH 

9 

7 

29 

45 

S 

8 

0 

9 

17 

Total 

62 

19 

130 

211 


84 


















































































































Writing Conventions 



R-WISE 

WRITE 

CONTROL 

TOTAL 

NH 

38 

19 

89 

146 

NSH 

17 

0 

25 

42 

S 

7 

0 

16 

23 

Total 

62 

19 

130 

211 


2.3 Spring Pass/Fail Data 



R-WISE 

WRITE 

CONTROL 

TOTAL 

Pass 

23 

4 

37 

64 

Fail 

39 

15 

93 

147 

Total 

62 

19 

130 

211 


2.4 Spring Score Data 



R-WISE 

WRITE 

CONTROL 

TOTAL 

2 

0 

0 

7 

7 

3 

12 

6 

30 

48 

4 

27 

9 

56 

92 

Total 

39 

15 

93 

147 


2.5 Spring Subscore Data 


Content Organization 



R-WISE 

WRITE 


HMlTlIliKlIHB 

NH 

27 

11 

78 

116 

NSH 

12 

4 

15 

31 

S 

0 

0 

0 

0 

Total 

39 

15 

93 

147 


Language 



R-WISE 

WRITE 



NH 

23 

11 

53 

87 

NSH 

9 

4 

24 

37 

S 

7 

0 

16 

23 

Total 

39 

15 

93 

147 
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Writing Conventions 



R-WISE 

WRITE 

CONTROL 

TOTAL 

NH 

21 

10 

54 

85 

NSH 

7 

2 

20 

29 

S 

11 

3 

19 

33 

Total 

39 

15 

93 

147 
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1.0 BACKGROUND 


Several artificial intelligence-based software tutoring programs have been developed by the 
US Air Force's Armstrong Laboratory to aid in teaching of the fundamental skills of 
writing (R-WISE), science (ISIS), and dgebra word problem solving (WPS). Each of 
these tutors has been field tested at Dayton Dunbar High School and Trotwood-Madison 
High School as well as several other sites across the nation. The design of the R-WISE 
and WPS tutors suggests they may help students pass the Ohio Proficiency Test (OPT). 
Through correlation of student usage of the tutor with OPT results, local researchers hoped 
to determine if the tutors were indeed an advantageous tool for preparation for this 
important exam. Of course, the tutors and Ohio Proficiency Test were not designed for use 
together. A lack of an improvement in OPT scores does not necessarily indicate that the 
students did not learn as a result of using the tutor, it could simply mean that the OPT is not 
the appropriate method of measurement for the tutor. 


2.0 THE WORD PROBLEM SOT WING TIJTOR/OPT ANALYSIS 

This analysis includes data collected from Trotwood-Madison High School during the 
1993-1994 and 1994-1995 school years. 


2.1 THE WORD PROBLEM SOLVING TUTOR 

The WPS tutor is divided into a series of modules which address various topics. These 
modules were designed to be appropriate for the class material covered in a general 9th 
grade Algebra course. 
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2.2 THE MATH SECTION OF THE OPT 


The Ohio Department of Education has established a series of strands and learning 
outcomes which the math section of the Ohio Proficiency Test has been designed to 
measure. These strands and learning outcomes are listed in Table A3-1 and identified as to 
whether there are WPS tutor modules which address similar topics. Additional information 
about the mathematics learning outcomes is available in information published by the Ohio 
Department of Education. 

Examples of test results from the math section of the Ohio Proficiency Test are given in 
Table A3-2. Each section of the Ohio Proficiency Test is scored on three levels. The first 
level is a pass/fail marking. A student who passes the test is not given any further scoring 
(i.e. score or subscores are omitted), and only a passing mark is returned to the school. A 
student who fails the math section of the test is then given a general score which can range 
from 0 to 199 given to three significant figures (i.e. 176, 180,...). The student is then 
given a subscore ranking in each of the five following strand areas: Arithmetic, 
Measurement, Geometry, Data Analysis, and Algebra. The possible subscore rankings are 
+, *, or - as defined in Table A3-2. 

Through analysis of the goals and directives of the WPS tutor and the Ohio Proficiency 
Test, it was determined by the researchers that the WPS tutor did not directly address the 
issues evaluated in the Data Analysis subscore area. Therefore, the tutor would not be 
expected to increase student performance in this area. The other areas, however, were 
believed to be addressed by the tutor and should be impacted by tutor usage. 


2.3 STUDY POPULATIONS 

The study population included only those students who failed the math section of the Ohio 
Proficiency Test in the fall of the year of study and for whom data was available for the 
spring test of that same year. Data were collected for the 1993-1994 school year and the 
1994-1995 school year. The sample size for each population group and year are given in 
Table A3-3. 
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Table A3-1: WPS tutor modules and OPT mathematics strands. 



PT MATHEMATICS STRANDS 


Arithmetic 


1. Compute with whole numbers, fractions, and decimals 

2. Compare, order, and determine equivalence of 
fractions, decimals, percents, whole numbers, and integers 

3. Solve and use proportions 

4. Round numbers to the nearest thousand, hundred, ten, 
one, tenth, and hundredth 

5. Solve problems and make applications involving 
percentages 


Measurement 

6. Select and compute with appropriate standard or metric 
units to measure length, area, volume, angles, weight, 
capacity, time, temperature, and money 

7. Convert, compare, and compute with common units of 
measure within the same measurement system 

8. Read the scale on a measurement device to the nearest 
mark and make interpolations where appropriate 


Geometry 

9. Recognize, classify, and use characteristics of lines and 
simple two-dimensional figures 

10. Find the perimeters (circumference) and areas of 
polygons (circles) 

11. Find surface areas and volumes of rectangular solids 


Data Analysis 

12. Read, interpret, and use tables, charts, maps, and 
graphics to identify patterns, note trends, and draw 
conclusions 

13. Use elementary notions of probability 

14. Compute averages 


Algebra 

15. Solve simple number sentences and use formulas 

16. Evaluate dgebraic expressions (simple substitutions) 


1 FOCUS OF 

WPS 

TUTOR 

1. 

Yes 

2. 

No 

3. 

Yes 

4. 

Yes 

5. 

Yes 

6. 

No 

7. 

Yes 

8. 

No 

9. 

No 

10. 

Yes 

11. 

Yes 

12. 

No 

13. 

No 

14. 

No 

15. 

Yes 

16. 

Yes 
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Table A3-2: Sample results for the math section of the Ohio Proficiency Test. 


NAME 

PASS 

or 

FAIL 

SCORE 

MEASURE¬ 

MENT 

ARITH¬ 

METIC 

GEOM¬ 

ETRY 

DATA 

ANALYSIS 

ALGE¬ 

BRA 

Larry 

Student 

Pass 







Mary 

Student 

Fail 

182 

+ 

- 

- 

* 

- 

Carry 

Student 

Fail 

176 

- 

- 

- 

+ 

* 

Berry 

Student 

Fail 

196 

+ 

+ 


* 



- = Performance lower than expected of students at the standard 
* = Performance approximately the same as expected of students at the standard 
+ = Performance higher than expected of students at the standard 


Table A3-3: Number of students included in the WPS data analysis for each group and 
year. 



WPS 

[ Control 

Total 

Practical 

Algebra 

Workshop 

Practical 

Algebra 

Workshop 


55 

20 

0 

0 

75 

94-95 

School 

Year 

37 

72 

53 

52 

214 

Total 

92 

92 

53 

52 

289 


2.3.1 CONTROL 


The control population included students who were enrolled in an algebra class which did 
not use the Air Force-developed software designed to enhance word problem solving 
proficiency. During the first year of the study, no control population was available at 
Trotwood-Madison High School. Regardless, the control population would include 
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students who were operating at class level, above class level, and below class level. For 
the purposes of this study, the control population was assumed to be equivalent to the 
treatment population. The accuracy of this assumption can be validated by comparison of 
fall scores and subscores on the Ohio Proficiency Test. This assumption is further 
validated by the fact that the study population was limited to students who failed the math 
portion of the Ohio Proficiency Test during the fall of the year in question. 


2.3.2 WPS 

The WPS population included students from Trotwood-Madison High Schools who were 
enrolled in either a Practical Algebra or Workshop class which used the WPS tutor. This 
group represents the experimental population in this study. 


2.4 DATA ANALYSIS 

In all cases, a Chi-Square analysis with a confidence interval of 5% was used to determine 
the statistical significance of the data. This indicates that if the numbers are statistically 
significant, then there is at least a 95% certainty that the same statistics would be observed 
in other, equivalent, populations. Tabular representations of all of the data presented are 
included in the appendix. 


2.4.1 FALL 


2.4.1.1 PASS/FAIL ANALYSIS 

Due to the fact that the study population was limited to students who failed the OPT in the 
fall, no comparison can be made in this area. 

2.4.1.2 SCORE ANALYSIS 

Due to the nearly continuous spread in the scores for the math section of the OPT, this 
analysis is complex and is not available at this time. 
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2.4.1.3 


SUBSCORE ANALYSIS 


Data for the subscore analysis are shown in Figure A3-1 for the Practical Algebra classes 
and Figure A3-2 for the Workshop classes. Chi-Squared analysis of these fall subscores 
showed no statistically significant variance in the groups for either the Practical Algebra or 
Workshop classes. This indicates that the treatment and control groups were statistically 
equivalent at the beginning of the year. This enhances the validity of our comparison 
between these two groups. 



Figure A3-1; Fall subscore for Practical Algebra classes. The differences between the 
control group and WPS group are not statistically significant. 















Figure A3-2; Fall subscore for Workshop classes. The differences between the control 
group and the WPS groups are not statistically significant. 


2.4.2 SPRING 

2.4.2.1 PASS/FAIL ANALYSIS 

This measure of the data gives the clearest representation of the effectiveness of the tutor at 
helping students pass the OPT. The goal of each student is to pass the test. An increase in 

score may suggest an increase in ability, but without passing the test the students and 
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schools do not measure any great improvement. Figure A3-3 displays the percentage of 
Practical Algebra students in the WPS and control groups passing the test during the spring 
of the year of study. These differences are not statistically significant. 



WPS Control 


Figure A3-3: Passing percent for Practical Algebra classes. The difference between the 
groups is not statistically significant. 


This suggests that use of the WPS tutor had no effect on these students passing the math 
section of the OPT. The lack of a difference in the passing rates could easily be attributed 
to other variables such as teacher differences, student population differences, and other 
non-controlled variables. 

The results for the Workshop course were statistically significant and are shown in Figure 
A3-4. These results suggest that the tutor may have lowered the chances of the students 
being able to pass the OPT. This could be a result of loss of class time due to use of the 
tutor, if indeed the tutor does not help. However, this could also simply be a result of 
other differences between the treatment and control populations and could have no basis in 
tutor usage at all. 
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Figure A3-4: Passing percent for the Workshop classes. The difference between the 
groups is statistically significant. 


2.4.2.2 SCORE ANALYSIS 

Due to the nearly continuous spread in the scores for the math section of the OPT, this 
analysis is complex and is not available at this time. 


2.4.2.3 SUBSCORE ANALYSIS 

The Chi-Squared analysis of spring subscores for the Practical Algebra classes showed a 
statistically significant variance in the subscore categories of Data Analysis and Algebra 
(Figure A3-5). The other subscore categories did not show any significant difference 
between the treatment and the control populations. The area of Data Analysis was not 
believed to be addressed by the tutor. However, students who used the tutor appeared to 
do better in this subscore area. This could indicate that either the initial tutor goals were not 
properly identified, or the OPT is measuring another difference between the study 
populations. The control population appeared to perform better in the area of Algebra 
which should have been addressed by the tutor. This again suggests that either the tutor is 
not effective, or the OPT is not a good measure of the effect of the tutor. 
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The Chi-Squared analysis of the spring subscore data for the Workshop classes showed the 
two groups to be statistically equivalent in all areas except Arithmetic and Geometry. Both 
of these areas were believed to be addressed by the tutor. However, in both cases the 
control population appeared to perform better than the WPS population (Figure A3-6). 

This suggests, again, that either the tutor is not an effective method for teaching this 
material, or the OPT is not a good measurement technique for learning associated with the 
tutor. 
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Figure A3-5: Spring subscore for Practical Algebra classes. Subscore rankings are: 
performed lower than expected for students at the standard (-), performed approximately 
the same as expected of students at the standard (*), or performed higher than expected of 
students at the standard (-I-). 
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Figure A3-6: Spring subscore by group for the Workshop classes. Subscore rankings are: 
performed lower than expected for students at the standard (-), performed approximately 
the same as expected of students at the standard (*), or performed higher than expected of 
students at the standard (+). 


2.5 RESULTS OF WPS TUTOR ANALYSIS 


Although a lot of data were collected and analyzed, the most relevant question to be 

answered relates to the effectiveness of use of the tutor to aid students with passing the 
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OPT. The fact that the study population was limited to students who failed the math 
portion of the OPT in the fall, and that the comparison of subscores between groups in the 
fall suggests that the groups were statistically equivalent at the beginning of the school 
year, suggests that this statistical comparison between students who did and did not use the 
tutor should be valid. It does not, however, mean that the OPT is a valid test to measure 
student learning with the WPS tutor. It will simply be an indication of whether students 
who used the tutor performed better on the OPT. 

The lack of any statistical significance in the passing rate of Practical Algebra students who 
used the WPS tutor over those who did not indicates that the WPS software may not be 
effective at helping students pass this important exam. The significant improvement in 
passing rate for students who did not use the tutor in the Workshop classes over students in 
Workshop classes which did use the tutor could suggest that the tutor is not an appropriate 
teaching aid for this group of students. It could also, however, simply be a reflection of 
the inability of OPT data to reflect learning accomplished through the use of the tutor. 

The inconsistency of results between the Practical Algebra and Workshop classes (Table 
A3-4) could be a result of other differences in the study population and, therefore, their 
ability to learn from the WPS tutor, or it could be a result of the inability of the OPT to 
effectively measure the material which the students learned as a result of exposure to the 
tutor. 


Table A3-4: Group which performed the best on a given area of the math section of the 
OPT. 


Measurement 


Arithmetic 


Geometry 


Algebra 


L_ Practical Algebra 

1 Fall 


1 




HS 

uivalent 


WPS 


Control 


























Of course, other factors not controlled by the researchers can always affect the observed 
results. Differences in student populations, teacher experience and style, and mid-year 
updates of the tutors themselves could all play a part by affecting the data analysis. 


3.0 THE R-WISE TUTOR/OPT ANALYSIS 

This analysis includes data collected from Dayton Belmont High School and Trotwood- 
Madison High School during the 1993-1994 and 1994-1995 school years. 


3.1 THE R-WISE TUTOR 

The R-WISE tutor (Reading and Writing in a Supportive Environment) is divided into a 
series of tools which address various topics. Through observation and use of the tutor and 
analysis of some of the literature provided by Armstrong Laboratory on the objectives of 
the tutor in correlations were made between the writing characteristics and learning 
outcomes defined by the Ohio Department of Education for the OPT and the tutor itself. 
This comparison is shown in Table A3-5. 


3.2 THE WRITING SECTION OF THE OPT 

The Ohio Department of Education has established a series of characteristics and learning 
outcomes which the writing section of the Ohio Proficiency Test has been designed to 
measure. Table A3-5 lists the writing characteristics and related learning outcomes as 
defined by the Ohio Department of Education. 

Examples of possible test results from the writing section of the Ohio Proficiency Test are 
given in Table A3-6. The writing section of the OPT is scored on three levels. The first 
level is a pass/fail marking. A student who passes the test is not given any further scoring 
(i.e. score or subscores are omitted), and only a passing mark is returned to the school. A 
student who fails the writing section of the test is then given a general score which can 
range from 0 to 4.5 given in two significant figures (i.e. 3.5, 4.0,...). The student is then 
given a subscore ranking in each of the three following characteristic areas; 
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Content/Organization, Language, and Writing Conventions. The possible subscore 
rankings are: satisfactory, needs some help, or needs help. 


Table A3-5: OPT writing characteristics and R-WISE tutor objectives. 


OPT WRITING CHARACTEWSTICS 

FOCUS OF 

R-WISE TUTOR 

Content/Orsanization 


1. Conveys a message related to the prompt 

1. Yes 

2. Includes supporting ideas or examples 

2. Yes 

3. Follows a logical order 

3. Yes 

4. Conveys a sense of completeness 

4. Yes 

Language 


5. Exhibits word choice appropriate to the audience. 

5. Yes 

purpose, and subject 


6. Includes clear language 

6. No 

Writing Conventions 


7. Contains complete sentences'and may contain 

7. No 

purposeful fragments 


8. Exhibits subject-verb agreement 

8. No 

9. Contains standard forms of verbs and nouns 

9. No 

10. Exhibits appropriate punctuation 

10. No 

11. Exhibits appropriate capitalization 

11. No 

12. Contains correct spelling 

12. No 

13. Is legible 

13. No 
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Table A3-6: Sample results for the writing section of the Ohio Proficiency Test. 


NAME 

PASS 

or 

FAIL 

SCORE 

ORGANIZATION 

LANGUAGE 

WRITING 

CONVENTIONS 

Larry 

Student 

Pass 





Mary 

Student 

Fail 

4.0 

NH 

NSH 

s 

Carry 

Student 

Fail 

3.5 

NSH 

NH 

NH 

Berry 

Student 

Fail 

4.5 

NH 

NH 

S 


S = Satisfactory 

NSH = Needs Some Help 

NH = Needs Help 


Through analysis of the goals and directives of the R-WISE tutor and the Ohio Proficiency 
Test, it was determined by the researchers that the R-WISE tutor did not directly address 
the issues evaluated in the Writing Conventions subscore area. Therefore, the tutor would 
not be expected to increase student performance in this area. The other areas, however, 
were believed to be addressed by the tutor and should be affected by tutor usage. 


3.3 STUDY POPULATIONS 

The study population included only those students who failed the writing section of the 
Ohio Proficiency Test in the fall of the year of study and for whom data were available for 
the spring test of that same year. Data were collected for the 1993-1994 school year and 
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the 1994-1995 school year. The sample size for each population group and year is given in 
Table A3-7. 


Table A3-7: Number of students included in R-WISE data analysis for each group and 
year. 



R-WISE 

T-M 

WRITE 

T-M 

Control 

T-M 

Total 

93-94 

School 

Year 

44 

12 

42 

98 

94-95 

School 

Year 

111 

0 

0 

111 

Total 

154 

V:'.12 

42 

'".' 20 ' 


3.3.1 CONTROL 

The control population included students who were not exposed to any Air Force- 
developed software designed to enhance writing proficiency. The majority of this 
population were students in the ninth grade at Trotwood-Madison High School during the 
93-94 school year who were not enrolled in a ninth grade English class which used the 
tutor. Because class rosters were not available for classes which did not use the tutor, the 
remaining 9th grade population was used. As a result, students who were enrolled in a 
class other than a standard 9th grade English class would be included in this population 
group. This could include students who were in honors-level or remedial-level courses. In 
either case the control population would include students who were operating at class level, 
above class level, and below class level. For the purposes of this study, the control 
population was assumed to be equivalent to the treatment population. The accuracy of this 
assumption can be validated by comparison of fall scores and subscores on the Ohio 
Proficiency Test. This assumption is further validated by the fact that the study population 
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was limited to students who failed the writing portion of the Ohio Proficiency Test during 
the fall of the year in question. 


3.3.2 WRITE 

The WRITE population included students who were enrolled in a standard 9th grade 
English class which used a simple word processor to assist in writing. The software did 
not contain any of the tutor assistance available in the more advanced version of the tutor. 
This group was used to study the effect of technology usage alone compared to the use of 
the more advanced tutor. 


3.3.3 R-WISE 

The R-WISE population included students who were enrolled in a standard 9th grade 
English class which used the advanced R-WISE tutor. This group represents the true 
experimental population in this study. 


3.4 DATA ANALYSIS 

In all cases, a Chi-Square analysis with a confidence interval of 5% was used to determine 
the statistical significance of the data. This indicates that if the numbers are statistically 
significant, then there is at least a 95% certainty that the same statistics would be observed 
in other, equivalent, populations. Tabular representations of all of the data presented are 
included in the appendix. 

3.4.1 FALL 


3.4.1.1 PASS/FAIL ANALYSIS 

Due to the fact that the study population was limited to students who failed the OPT in the 
fall, no comparison can be made in this area. 
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3.4.1.2 


SCORE ANALYSIS 


The Chi-Squared analysis of fall scores showed a statistically significant variance in the 
data; observation of Figure A3-7 shows that the WRITE group performed the best, 
followed by the R-WISE and control groups. 



Figure A3-7: Fall score by group for R-WISE analysis. 


3.4.1.3 SUBSCORE ANALYSIS 

Comparison of the fall subscores between groups showed that the groups were statistically 
equivalent in all of the areas. This indicates that the groups were equivalent at the 
beginning of the study and adds validity to the comparison of these groups at the end of the 
year. 


3.4.2 SPRING 

3.4.2.1 PASS/FAIL ANALYSIS 

This measure of the data gives the clearest representation of the effectiveness of the tutor at 
helping students pass the OPT. The goal of each student is to pass the test. An increase in 
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score may suggest an increase in ability, but without passing the test the students and 
schools do not measure any great improvement. Figure A3-8 displays the percentage of 
students in each group passing the test during the spring of the year of study. These 
differences are statistically significant and may represent an improvement as a result of 
exposure to the R-WISE tutor. 
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Figure A3-8: Passing percent by group for R-WISE analysis. 


The increase in the passing rate for the Control group over the WRITE group suggests that 
time spent in the computer lab and away from other organized learning activities may have 
been detrimental to the students. There is a significant increase measured, however, as a 
result of the features present in the R-WISE version of the software. Again, care must be 
taken in comparison of the data, particularly in the situation of the Control group and the 
WRITE group. The difference in the passing rates is small and could easily be attributed to 
other variables such as teacher differences, curriculum variations, student population 
differences, and other non-controlled variables. 


3.4.2.2 SCORE ANALYSIS 


The Chi-Squared analysis of spring scores showed no statistically significant variance in 
the data. The groups scores were statistically equivalent in the fall and remained that way 







in the spring. Therefore, this analysis does not suggest that the R-WISE tutor had any 
measurable affect on student performance. 


3.4.2.3 SUBSCORE ANALYSIS 

Comparison of spring subscores by group showed statistical significance only in the area 
of Content Organization (see Figure A3-9). 



Figure A3-9: Spring subscore analysis by group for R-WISE analysis. 


Ill 















Comparison of the fall and spring results is presented in Table A3-8. In the fall subscore 
comparison of the three groups, they were statistically equivalent in all areas. After 
exposure of the R-WISE and WRITE groups to computer-based training, the groups were 
statistically equivalent in all areas except Content Organization. 


Table A3-8: Group which performed the best on a given area of the writing section of the 
OPT. 



Fall Exam 

Spring Exam 

Content Organization 

Equivalent 

R-WISE 

Language 

Equivalent 

Equivalent 

Writing Conventions 

Equivalent 

Equivalent 


In the area of Content Organization, the R-WISE group performed the best in the spring. 
This suggests that the students using the R-WISE tutor were able to improve their 
performance more than students who were not exposed to the tutor. This suggests that the 
R-WISE tutor is able to improve student performance on the OPT. 


3.5 RESULTS OF R-WISE DATA ANALYSIS 

Although a lot of data were collected and analyzed, the most relevant question to be 
answered relates to the effectiveness of use of the tutor to aid students with passing the 
OPT. The fact that the study population was limited to students who failed the writing 
portion of the OPT in the fall, and that the comparison of subscores between groups in the 
fall showed no significant difference between groups, suggests that the groups were 
statistically equivalent at the beginning of the school year. The increased passing rate of the 
students who used the R-WISE tutor over those who did not is significant. This strongly 
indicates that the R-WISE software may be effective at helping students pass this important 
exam. The greatest help appears to be in the area of Content Organization which is the area 
most strongly addressed by the tutor. 
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Of course, other factors not controlled by the researchers can always affect the observed 
results. Differences in class curriculums, student populations, teacher experience and 
style, and mid-year updates of the tutors themselves could all play a part in affecting the 
data analysis. 

However, the data collected to date suggest that the R-WISE software can increase the 
passing rate of students on the writing portion of the OPT by about 20%. 


4.0 CONCLUSIONS 

No significant difference was measured in the performance of Practical Algebra students as 
a result of exposure to the tutor. The performance of Workshop students who were not 
exposed to the tutor was actually higher than those who were exposed to it. These results 
suggest that the WPS may not be effective at helping students pass the math section of the 
Ohio Proficiency Test. 

The results of the R-WISE tutor are more promising. Students who used the tutor had a 
20% higher passing rate over students who were exposed to traditional instruction alone. 
These results are encouraging. However, other factors such as student variations, teacher 
differences, cuiriculum differences, and other uncontrolled factors could have an effect on 
these results and must always be considered. 
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APPENDIX - TABULATED DATA 


1.0 WPS TUTOR 

1.1 Practical Algebra 

1.1.1 Fall Subscore Data 


Measurement 



WPS 

Control 

Total 


64 

41 

105 


25 

11 

36 

+ 

3 

1 

4 

Total 

92 

53 

145 


Arithmetic 



WPS 

Control 

Total 

- 

65 

39 

104 

* 

23 

13 

36 

+ 

4 

1 

5 

Total 

92 

53 

145 


Geometry 



WPS 

Control 

Total 

- 

27 

13 

40 

* 

54 

32 

86 

+ 

11 

8 

19 

Total 

92 

53 

145 


Data Analysis 



WPS 

Control 

Total 


48 

27 

75 

* 

33 

22 

55 

+ 

11 

4 

15 

Total 

92 

53 

145 
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Algebra 



WPS 

Control 

Total 


46 

34 

80 

* 

41 

17 

58 

+ 

5 

2 

7 

Total 

92 

53 

145 


1.1.2 Spring Pass/Fail Data 



WPS 

Control 

Total 

Pass 

12 

13 

25 

Fail 

80 

40 

120 

Total 

92 

53 

145 


1.1.3 Spring Subscore Data 

Measurement 



WPS 

Control 

Total 

- 

48 

27 

75 

* 

31 

12 

43 

+ 

1 

1 

2 

Total 

80 

40 

120 


Arithmetic 



WPS 

Control 

Total 

- 

58 

31 

89 

si: 

20 

8 

28 

+ 

2 

1 

3 

Total 

80 

40 

120 


Geometry 



WPS 

Control 

Total 

- 

21 

15 

36 

* 

43 

21 

64 

+ 

16 

4 

20 

Total 

80 

40 

120 
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Data Analysis 



WPS 

Control 

Total 

- 

48 

33 

81 


29 

7 

36 

+ 

3 

0 

3 

Total 

80 

40 

120 


Algebra 



WPS 

Control 

Total 

- 

38 

13 

51 


40 

24 

64 

+ 

2 

3 

5 

Total 

80 

40 

120 


1.2 Workshop Classes 

1.2.1 Fall Subscore Data 


Measurement 



WPS 

Control 

Total 


78 

40 

118 


12 

12 

24 

+ 

2 

0 

2 

Total 

92 

52 

144 


Arithmetic 



WPS 

Control 

Total 

- 

78 

41 

119 


12 

10 

22 

+ 

2 

1 

3 

Total 

92 

52 

144 


Geometry 



WPS 

Control 

Total 

- 

43 

17 

60 


43 

26 

69 

+ 

6 

9 

15 

Total 

92 

52 

144 
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Data Analysis 



WPS 

Control 

Total 

- 

53 

28 

81 


37 

21 

58 

+ 

2 

3 

5 

Total 

92 

52 

144 


Algebra 



WPS 

Control 

Total 

- 

60 

33 

93 

* 

28 

18 

46 

+ 

4 

1 

5 

Total 

92 

52 

144 


1.2.2 Spring Pass/Fail Data 



WPS 

Control 

Total 

Pass 

13 

15 

28 

Fail 

74 

37 

111 

Total 

87 

52 

139 


1.2.3 Spring Subscore Data 


Measurement 



WPS 

Control 

Total 


49 

24 

73 

* 

23 

13 

36 

+ 

2 

0 

2 

Total 

74 

37 

111 


Arithmetic 



WPS 

Control 

Total 

- 

60 

19 

79 

* 

13 

16 

29 

+ 

1 

2 

3 

Total 

74 

37 

111 
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Geometry 



WPS 

Control 

Total 

- 

35 

12 

47 


34 

18 

52 

+ 

5 

7 

12 

Total 

74 

37 

111 


Data Analysis 



WPS 

Control 

Total 

— 

42 

24 

66 


31 

13 

44 

+ 

1 

0 

1 

Total 

lA 

37 

111 


Algebra 



WPS 

Control 

Total 

- 

35 

14 

49 


32 

22 

54 

+ 

7 

1 

8 

Total 

74 

37 

111 


2.0 R-WISE TUTOR 


2.1 Fall Score Data 



R-WISE 

WRITE 

CONTROL 

TOTAL 

2 

1 

1 

3 

5 

3 

27 

0 

10 

37 

4 

126 

11 

29 

166 

Total 

154 

12 

42 

208 


2.1 Fall Subscore Data 


Content Organization 



R-WISE 

WRITE 

CONTROL 

TOTAL 

NH 

137 

10 

39 

186 

NSH 

17 

2 

3 

22 

S 

0 

0 

0 

0 

Total 

154 

12 

42 

208 
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Language 



R-WISE 

WRITE 

CONTROL 

TOTAL 

NH 

92 

8 

28 

128 

NSH 

49 

4 

9 

62 

S 

13 

0 

5 

18 

Total 

154 

12 

42 

208 


Writing Conventions 



R-WISE 

WRITE 

CONTROL 

TOTAL 

NH 

76 

5 

21 

102 

NSH 

36 

4 

13 

53 

S 

42 

3 

8 

53 

Total 

154 

12 

42 

208 


2.3 Spring Pass/Fail Data 



R-WISE 

WRITE 

CONTROL 

TOTAL 

Pass 

61 

3 

34 

72 

Fail 

93 

9 

8 

136 

Total 

154 

12 

42 

208 


2.4 Spring Score Data 



R-WISE 

WRITE 

CONTROL 

TOTAL 

2 

3 

0 

5 

8 

3 

24 

1 

10 

25 

4 

66 

8 

19 

93 

Total 

93 

9 

34 

136 


2.5 Spring Subscore. Data 


Content Organization 



R-WISE 

WRITE 

CONTROL 


NH 

72 

9 

34 

115 

NSH 

19 

0 

0 

19 

S 

0 

0 

0 

0 

Total 

91 

9 

34 

134 


119 
















































































































Language 



R-WISE 

WRITE 


TOTAL 

NH 

49 

5 

27 

81 

NSH 

28 

3 

6 

37 

S 

14 

1 

1 

16 

Total 

91 ^ 

9 

34 

134 


Writing Conventions 



R-WISE 

WRITE 

CONTROL 

TOTAL 

NH 

43 

2 

20 

65 

NSH 

22 

2 

7 

31 

S 

26 

5 

7 

38 

Total 

91 

9 

34 

134 


120 




















































